__ __ / \ / \ ______ _ _ __ __ __ / /\ / /\ \ / __ \ | | | | / \ \ \ / / / / / / \ \ | | | | | | | | / /\ \ \ / \ \ \ \ / / | |__| | | |__| | / ____ \ | | \ \/ \ \/ / \_ ___/ \____/ /_/ \_\ |_| \__/ \__/ \ \__ \___\ by Red Hat Build, Store, and Distribute your Containers Startup timestamp: Thu Jul 6 07:18:13 UTC 2023 Running all default config services Installing extra certificates found in /quay-registry/conf/stack/extra_ca_certs directory 2023-07-06 07:18:14,596 INFO RPC interface 'supervisor' initialized 2023-07-06 07:18:14,597 CRIT Server 'unix_http_server' running without any HTTP authentication checking 2023-07-06 07:18:14,597 INFO supervisord started with pid 7 2023-07-06 07:18:15,599 INFO spawned: 'stdout' with pid 30 2023-07-06 07:18:15,600 INFO spawned: 'config-editor' with pid 31 2023-07-06 07:18:15,601 INFO spawned: 'quotaregistrysizeworker' with pid 32 2023-07-06 07:18:15,602 INFO spawned: 'quotatotalworker' with pid 33 2023-07-06 07:18:16,623 INFO success: stdout entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-07-06 07:18:16,623 INFO success: config-editor entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-07-06 07:18:16,623 INFO success: quotaregistrysizeworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-07-06 07:18:16,623 INFO success: quotatotalworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) config-editor stdout | time="2023-07-06T07:18:15Z" level=info msg="Using Operator Endpoint: http://quay-operator.quay-enterprise39.svc.cluster.local:7071" config-editor stdout | time="2023-07-06T07:18:15Z" level=warning msg="An error occurred loading TLS: No public key provided for HTTPS. Server falling back to HTTP." config-editor stdout | time="2023-07-06T07:18:15Z" level=info msg="Running the configuration editor with HTTP on port 8080 with username quayconfig" quotatotalworker stderr | /app/lib/python3.9/site-packages/magic.py:193: SyntaxWarning: "is" with a literal. Did you mean "=="? quotatotalworker stderr | if result is -1: quotaregistrysizeworker stderr | /app/lib/python3.9/site-packages/magic.py:193: SyntaxWarning: "is" with a literal. Did you mean "=="? quotaregistrysizeworker stderr | if result is -1: quotaregistrysizeworker stdout | 2023-07-06 07:18:17,885 [32] [DEBUG] [workers.worker] Scheduling worker. quotaregistrysizeworker stdout | 2023-07-06 07:18:17,885 [32] [INFO] [apscheduler.scheduler] Scheduler started quotaregistrysizeworker stdout | 2023-07-06 07:18:17,885 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 07:18:17,885 [32] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added quotaregistrysizeworker stdout | 2023-07-06 07:18:17,886 [32] [DEBUG] [workers.worker] First run scheduled for 2023-07-06 07:18:34.887001 quotaregistrysizeworker stdout | 2023-07-06 07:18:17,938 [32] [INFO] [apscheduler.scheduler] Added job "QuotaRegistrySizeWorker._calculate_registry_size" to job store "default" quotaregistrysizeworker stdout | 2023-07-06 07:18:17,938 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 07:18:17,938 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:18:34.887001+00:00 (in 16.948434 seconds) quotaregistrysizeworker stdout | 2023-07-06 07:18:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 07:18:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:18:34 UTC)" (scheduled at 2023-07-06 07:18:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 07:18:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 07:18:34,888 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:19:34.887001+00:00 (in 59.999697 seconds) quotaregistrysizeworker stdout | 2023-07-06 07:18:34,897 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 07:18:34,897 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:19:34 UTC)" executed successfully quotatotalworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stderr | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stderr | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stderr | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stderr | self.send(msg) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stderr | self.connect() quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stderr | self.sock = self._create_connection( quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stderr | raise err quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stderr | sock.connect(sa) quotatotalworker stderr | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stderr | During handling of the above exception, another exception occurred: quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stderr | push_to_gateway( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stderr | handler( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stderr | response = self._open(req, data) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stderr | result = func(*args) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stderr | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stderr | raise URLError(err) quotatotalworker stderr | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:18:47,715 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stderr | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stderr | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stderr | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stderr | self.send(msg) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stderr | self.connect() quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stderr | self.sock = self._create_connection( quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stderr | raise err quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stderr | sock.connect(sa) quotatotalworker stderr | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stderr | During handling of the above exception, another exception occurred: quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stderr | push_to_gateway( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stderr | handler( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stderr | response = self._open(req, data) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stderr | result = func(*args) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stderr | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stderr | raise URLError(err) quotatotalworker stderr | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:19:17,748 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:19:34,888 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 07:19:34,888 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:20:34.887001+00:00 (in 59.998621 seconds) quotaregistrysizeworker stdout | 2023-07-06 07:19:34,888 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:20:34 UTC)" (scheduled at 2023-07-06 07:19:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 07:19:34,889 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 07:19:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 07:19:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:20:34 UTC)" executed successfully quotatotalworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stderr | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stderr | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stderr | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stderr | self.send(msg) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stderr | self.connect() quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stderr | self.sock = self._create_connection( quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stderr | raise err quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stderr | sock.connect(sa) quotatotalworker stderr | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stderr | During handling of the above exception, another exception occurred: quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stderr | push_to_gateway( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stderr | handler( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stderr | response = self._open(req, data) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stderr | result = func(*args) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stderr | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stderr | raise URLError(err) quotatotalworker stderr | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:19:47,780 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stderr | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stderr | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stderr | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stderr | self.send(msg) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stderr | self.connect() quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stderr | self.sock = self._create_connection( quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stderr | raise err quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stderr | sock.connect(sa) quotatotalworker stderr | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stderr | During handling of the above exception, another exception occurred: quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stderr | push_to_gateway( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stderr | handler( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stderr | response = self._open(req, data) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stderr | result = func(*args) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stderr | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stderr | raise URLError(err) quotatotalworker stderr | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:20:17,792 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:20:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 07:20:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:21:34.887001+00:00 (in 59.999525 seconds) quotaregistrysizeworker stdout | 2023-07-06 07:20:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:21:34 UTC)" (scheduled at 2023-07-06 07:20:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 07:20:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 07:20:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 07:20:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:21:34 UTC)" executed successfully quotatotalworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stderr | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stderr | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stderr | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stderr | self.send(msg) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stderr | self.connect() quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stderr | self.sock = self._create_connection( quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stderr | raise err quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stderr | sock.connect(sa) quotatotalworker stderr | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stderr | During handling of the above exception, another exception occurred: quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stderr | push_to_gateway( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stderr | handler( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stderr | response = self._open(req, data) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stderr | result = func(*args) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stderr | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stderr | raise URLError(err) quotatotalworker stderr | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:20:47,816 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stderr | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stderr | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stderr | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stderr | self.send(msg) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stderr | self.connect() quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stderr | self.sock = self._create_connection( quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stderr | raise err quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stderr | sock.connect(sa) quotatotalworker stderr | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stderr | During handling of the above exception, another exception occurred: quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stderr | push_to_gateway( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stderr | handler( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stderr | response = self._open(req, data) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stderr | result = func(*args) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stderr | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stderr | raise URLError(err) quotatotalworker stderr | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:21:17,848 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:21:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 07:21:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:22:34.887001+00:00 (in 59.999534 seconds) quotaregistrysizeworker stdout | 2023-07-06 07:21:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:22:34 UTC)" (scheduled at 2023-07-06 07:21:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 07:21:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 07:21:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 07:21:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:22:34 UTC)" executed successfully quotatotalworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stderr | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stderr | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stderr | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stderr | self.send(msg) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stderr | self.connect() quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stderr | self.sock = self._create_connection( quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stderr | raise err quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stderr | sock.connect(sa) quotatotalworker stderr | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stderr | During handling of the above exception, another exception occurred: quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stderr | push_to_gateway( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stderr | handler( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stderr | response = self._open(req, data) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stderr | result = func(*args) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stderr | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stderr | raise URLError(err) quotatotalworker stderr | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:21:47,860 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stderr | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stderr | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stderr | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stderr | self.send(msg) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stderr | self.connect() quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stderr | self.sock = self._create_connection( quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stderr | raise err quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stderr | sock.connect(sa) quotatotalworker stderr | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stderr | During handling of the above exception, another exception occurred: quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stderr | push_to_gateway( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stderr | handler( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stderr | response = self._open(req, data) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stderr | result = func(*args) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stderr | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stderr | raise URLError(err) quotatotalworker stderr | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:22:17,876 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:22:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 07:22:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:23:34.887001+00:00 (in 59.999515 seconds) quotaregistrysizeworker stdout | 2023-07-06 07:22:34,888 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:23:34 UTC)" (scheduled at 2023-07-06 07:22:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 07:22:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 07:22:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 07:22:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:23:34 UTC)" executed successfully quotatotalworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stderr | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stderr | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stderr | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stderr | self.send(msg) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stderr | self.connect() quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stderr | self.sock = self._create_connection( quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stderr | raise err quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stderr | sock.connect(sa) quotatotalworker stderr | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stderr | During handling of the above exception, another exception occurred: quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stderr | push_to_gateway( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stderr | handler( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stderr | response = self._open(req, data) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stderr | result = func(*args) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stderr | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stderr | raise URLError(err) quotatotalworker stderr | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:22:47,908 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stderr | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stderr | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stderr | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stderr | self.send(msg) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stderr | self.connect() quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stderr | self.sock = self._create_connection( quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stderr | raise err quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stderr | sock.connect(sa) quotatotalworker stderr | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stderr | During handling of the above exception, another exception occurred: quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stderr | push_to_gateway( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stderr | handler( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stderr | response = self._open(req, data) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stderr | result = func(*args) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stderr | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stderr | raise URLError(err) quotatotalworker stderr | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:23:17,940 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:23:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 07:23:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:24:34.887001+00:00 (in 59.999506 seconds) quotaregistrysizeworker stdout | 2023-07-06 07:23:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:24:34 UTC)" (scheduled at 2023-07-06 07:23:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 07:23:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 07:23:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 07:23:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:24:34 UTC)" executed successfully quotatotalworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stderr | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stderr | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stderr | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stderr | self.send(msg) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stderr | self.connect() quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stderr | self.sock = self._create_connection( quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stderr | raise err quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stderr | sock.connect(sa) quotatotalworker stderr | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stderr | During handling of the above exception, another exception occurred: quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stderr | push_to_gateway( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stderr | handler( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stderr | response = self._open(req, data) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stderr | result = func(*args) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stderr | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stderr | raise URLError(err) quotatotalworker stderr | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:23:47,972 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stderr | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stderr | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stderr | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stderr | self.send(msg) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stderr | self.connect() quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stderr | self.sock = self._create_connection( quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stderr | raise err quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stderr | sock.connect(sa) quotatotalworker stderr | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stderr | During handling of the above exception, another exception occurred: quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stderr | push_to_gateway( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stderr | handler( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stderr | response = self._open(req, data) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stderr | result = func(*args) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stderr | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stderr | raise URLError(err) quotatotalworker stderr | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:24:18,004 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:24:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 07:24:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:25:34.887001+00:00 (in 59.999514 seconds) quotaregistrysizeworker stdout | 2023-07-06 07:24:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:25:34 UTC)" (scheduled at 2023-07-06 07:24:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 07:24:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 07:24:34,895 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 07:24:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:25:34 UTC)" executed successfully quotatotalworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stderr | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stderr | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stderr | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stderr | self.send(msg) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stderr | self.connect() quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stderr | self.sock = self._create_connection( quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stderr | raise err quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stderr | sock.connect(sa) quotatotalworker stderr | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stderr | During handling of the above exception, another exception occurred: quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stderr | push_to_gateway( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stderr | handler( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stderr | response = self._open(req, data) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stderr | result = func(*args) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stderr | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stderr | raise URLError(err) quotatotalworker stderr | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:24:48,036 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stderr | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stderr | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stderr | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stderr | self.send(msg) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stderr | self.connect() quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stderr | self.sock = self._create_connection( quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stderr | raise err quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stderr | sock.connect(sa) quotatotalworker stderr | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stderr | During handling of the above exception, another exception occurred: quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stderr | push_to_gateway( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stderr | handler( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stderr | response = self._open(req, data) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stderr | result = func(*args) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stderr | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stderr | raise URLError(err) quotatotalworker stderr | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:25:18,068 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:25:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 07:25:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:26:34.887001+00:00 (in 59.999546 seconds) quotaregistrysizeworker stdout | 2023-07-06 07:25:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:26:34 UTC)" (scheduled at 2023-07-06 07:25:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 07:25:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 07:25:34,895 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 07:25:34,895 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:26:34 UTC)" executed successfully quotatotalworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stderr | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stderr | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stderr | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stderr | self.send(msg) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stderr | self.connect() quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stderr | self.sock = self._create_connection( quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stderr | raise err quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stderr | sock.connect(sa) quotatotalworker stderr | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stderr | During handling of the above exception, another exception occurred: quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stderr | push_to_gateway( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stderr | handler( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stderr | response = self._open(req, data) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stderr | result = func(*args) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stderr | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stderr | raise URLError(err) quotatotalworker stderr | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:25:48,100 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stderr | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stderr | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stderr | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stderr | self.send(msg) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stderr | self.connect() quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stderr | self.sock = self._create_connection( quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stderr | raise err quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stderr | sock.connect(sa) quotatotalworker stderr | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stderr | During handling of the above exception, another exception occurred: quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stderr | push_to_gateway( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stderr | handler( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stderr | response = self._open(req, data) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stderr | result = func(*args) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stderr | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stderr | raise URLError(err) quotatotalworker stderr | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:26:18,117 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:26:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 07:26:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:27:34.887001+00:00 (in 59.999490 seconds) quotaregistrysizeworker stdout | 2023-07-06 07:26:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:27:34 UTC)" (scheduled at 2023-07-06 07:26:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 07:26:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 07:26:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 07:26:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:27:34 UTC)" executed successfully quotatotalworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stderr | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stderr | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stderr | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stderr | self.send(msg) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stderr | self.connect() quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stderr | self.sock = self._create_connection( quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stderr | raise err quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stderr | sock.connect(sa) quotatotalworker stderr | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stderr | During handling of the above exception, another exception occurred: quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stderr | push_to_gateway( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stderr | handler( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stderr | response = self._open(req, data) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stderr | result = func(*args) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stderr | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stderr | raise URLError(err) quotatotalworker stderr | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:26:48,149 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stderr | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stderr | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stderr | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stderr | self.send(msg) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stderr | self.connect() quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stderr | self.sock = self._create_connection( quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stderr | raise err quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stderr | sock.connect(sa) quotatotalworker stderr | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stderr | During handling of the above exception, another exception occurred: quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stderr | push_to_gateway( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stderr | handler( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stderr | response = self._open(req, data) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stderr | result = func(*args) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stderr | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stderr | raise URLError(err) quotatotalworker stderr | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:27:18,181 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:27:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 07:27:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:28:34.887001+00:00 (in 59.999518 seconds) quotaregistrysizeworker stdout | 2023-07-06 07:27:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:28:34 UTC)" (scheduled at 2023-07-06 07:27:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 07:27:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 07:27:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 07:27:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:28:34 UTC)" executed successfully quotatotalworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stderr | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stderr | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stderr | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stderr | self.send(msg) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stderr | self.connect() quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stderr | self.sock = self._create_connection( quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stderr | raise err quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stderr | sock.connect(sa) quotatotalworker stderr | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stderr | During handling of the above exception, another exception occurred: quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stderr | push_to_gateway( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stderr | handler( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stderr | response = self._open(req, data) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stderr | result = func(*args) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stderr | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stderr | raise URLError(err) quotatotalworker stderr | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:27:48,213 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stderr | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stderr | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stderr | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stderr | self.send(msg) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stderr | self.connect() quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stderr | self.sock = self._create_connection( quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stderr | raise err quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stderr | sock.connect(sa) quotatotalworker stderr | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stderr | During handling of the above exception, another exception occurred: quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stderr | push_to_gateway( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stderr | handler( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stderr | response = self._open(req, data) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stderr | result = func(*args) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stderr | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stderr | raise URLError(err) quotatotalworker stderr | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:28:18,240 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:28:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 07:28:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:29:34.887001+00:00 (in 59.999500 seconds) quotaregistrysizeworker stdout | 2023-07-06 07:28:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:29:34 UTC)" (scheduled at 2023-07-06 07:28:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 07:28:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 07:28:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 07:28:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:29:34 UTC)" executed successfully quotatotalworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stderr | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stderr | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stderr | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stderr | self.send(msg) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stderr | self.connect() quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stderr | self.sock = self._create_connection( quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stderr | raise err quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stderr | sock.connect(sa) quotatotalworker stderr | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stderr | During handling of the above exception, another exception occurred: quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stderr | push_to_gateway( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stderr | handler( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stderr | response = self._open(req, data) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stderr | result = func(*args) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stderr | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stderr | raise URLError(err) quotatotalworker stderr | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:28:48,273 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stderr | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stderr | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stderr | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stderr | self.send(msg) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stderr | self.connect() quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stderr | self.sock = self._create_connection( quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stderr | raise err quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stderr | sock.connect(sa) quotatotalworker stderr | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stderr | During handling of the above exception, another exception occurred: quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stderr | push_to_gateway( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stderr | handler( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stderr | response = self._open(req, data) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stderr | result = func(*args) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stderr | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stderr | raise URLError(err) quotatotalworker stderr | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:29:18,305 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:29:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 07:29:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:30:34.887001+00:00 (in 59.999460 seconds) quotaregistrysizeworker stdout | 2023-07-06 07:29:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:30:34 UTC)" (scheduled at 2023-07-06 07:29:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 07:29:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 07:29:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 07:29:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:30:34 UTC)" executed successfully quotatotalworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stderr | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stderr | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stderr | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stderr | self.send(msg) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stderr | self.connect() quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stderr | self.sock = self._create_connection( quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stderr | raise err quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stderr | sock.connect(sa) quotatotalworker stderr | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stderr | During handling of the above exception, another exception occurred: quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stderr | push_to_gateway( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stderr | handler( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stderr | response = self._open(req, data) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stderr | result = func(*args) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stderr | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stderr | raise URLError(err) quotatotalworker stderr | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:29:48,337 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stderr | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stderr | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stderr | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stderr | self.send(msg) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stderr | self.connect() quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stderr | self.sock = self._create_connection( quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stderr | raise err quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stderr | sock.connect(sa) quotatotalworker stderr | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stderr | During handling of the above exception, another exception occurred: quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stderr | push_to_gateway( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stderr | handler( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stderr | response = self._open(req, data) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stderr | result = func(*args) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stderr | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stderr | raise URLError(err) quotatotalworker stderr | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:30:18,370 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:30:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 07:30:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:31:34.887001+00:00 (in 59.999538 seconds) quotaregistrysizeworker stdout | 2023-07-06 07:30:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:31:34 UTC)" (scheduled at 2023-07-06 07:30:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 07:30:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 07:30:34,895 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 07:30:34,895 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:31:34 UTC)" executed successfully quotatotalworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stderr | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stderr | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stderr | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stderr | self.send(msg) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stderr | self.connect() quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stderr | self.sock = self._create_connection( quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stderr | raise err quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stderr | sock.connect(sa) quotatotalworker stderr | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stderr | During handling of the above exception, another exception occurred: quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stderr | push_to_gateway( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stderr | handler( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stderr | response = self._open(req, data) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stderr | result = func(*args) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stderr | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stderr | raise URLError(err) quotatotalworker stderr | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:30:48,384 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stderr | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stderr | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stderr | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stderr | self.send(msg) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stderr | self.connect() quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stderr | self.sock = self._create_connection( quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stderr | raise err quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stderr | sock.connect(sa) quotatotalworker stderr | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stderr | During handling of the above exception, another exception occurred: quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stderr | push_to_gateway( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stderr | handler( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stderr | response = self._open(req, data) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stderr | result = func(*args) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stderr | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stderr | raise URLError(err) quotatotalworker stderr | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:31:18,397 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:31:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 07:31:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:32:34.887001+00:00 (in 59.999555 seconds) quotaregistrysizeworker stdout | 2023-07-06 07:31:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:32:34 UTC)" (scheduled at 2023-07-06 07:31:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 07:31:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 07:31:34,895 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 07:31:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:32:34 UTC)" executed successfully quotatotalworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stderr | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stderr | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stderr | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stderr | self.send(msg) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stderr | self.connect() quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stderr | self.sock = self._create_connection( quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stderr | raise err quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stderr | sock.connect(sa) quotatotalworker stderr | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stderr | During handling of the above exception, another exception occurred: quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stderr | push_to_gateway( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stderr | handler( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stderr | response = self._open(req, data) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stderr | result = func(*args) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stderr | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stderr | raise URLError(err) quotatotalworker stderr | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:31:48,424 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stderr | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stderr | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stderr | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stderr | self.send(msg) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stderr | self.connect() quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stderr | self.sock = self._create_connection( quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stderr | raise err quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stderr | sock.connect(sa) quotatotalworker stderr | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stderr | During handling of the above exception, another exception occurred: quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stderr | push_to_gateway( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stderr | handler( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stderr | response = self._open(req, data) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stderr | result = func(*args) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stderr | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stderr | raise URLError(err) quotatotalworker stderr | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:32:18,457 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:32:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 07:32:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:33:34.887001+00:00 (in 59.999527 seconds) quotaregistrysizeworker stdout | 2023-07-06 07:32:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:33:34 UTC)" (scheduled at 2023-07-06 07:32:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 07:32:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 07:32:34,895 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 07:32:34,895 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:33:34 UTC)" executed successfully quotatotalworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stderr | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stderr | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stderr | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stderr | self.send(msg) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stderr | self.connect() quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stderr | self.sock = self._create_connection( quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stderr | raise err quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stderr | sock.connect(sa) quotatotalworker stderr | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stderr | During handling of the above exception, another exception occurred: quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stderr | push_to_gateway( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stderr | handler( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stderr | response = self._open(req, data) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stderr | result = func(*args) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stderr | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stderr | raise URLError(err) quotatotalworker stderr | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:32:48,489 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stderr | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stderr | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stderr | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stderr | self.send(msg) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stderr | self.connect() quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stderr | self.sock = self._create_connection( quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stderr | raise err quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stderr | sock.connect(sa) quotatotalworker stderr | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stderr | During handling of the above exception, another exception occurred: quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stderr | push_to_gateway( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stderr | handler( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stderr | response = self._open(req, data) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stderr | result = func(*args) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stderr | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stderr | raise URLError(err) quotatotalworker stderr | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:33:18,521 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:33:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 07:33:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:34:34.887001+00:00 (in 59.999522 seconds) quotaregistrysizeworker stdout | 2023-07-06 07:33:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:34:34 UTC)" (scheduled at 2023-07-06 07:33:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 07:33:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 07:33:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 07:33:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:34:34 UTC)" executed successfully quotatotalworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stderr | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stderr | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stderr | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stderr | self.send(msg) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stderr | self.connect() quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stderr | self.sock = self._create_connection( quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stderr | raise err quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stderr | sock.connect(sa) quotatotalworker stderr | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stderr | During handling of the above exception, another exception occurred: quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stderr | push_to_gateway( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stderr | handler( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stderr | response = self._open(req, data) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stderr | result = func(*args) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stderr | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stderr | raise URLError(err) quotatotalworker stderr | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:33:48,554 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stderr | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stderr | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stderr | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stderr | self.send(msg) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stderr | self.connect() quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stderr | self.sock = self._create_connection( quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stderr | raise err quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stderr | sock.connect(sa) quotatotalworker stderr | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stderr | During handling of the above exception, another exception occurred: quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stderr | push_to_gateway( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stderr | handler( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stderr | response = self._open(req, data) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stderr | result = func(*args) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stderr | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stderr | raise URLError(err) quotatotalworker stderr | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:34:18,586 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:34:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 07:34:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:35:34.887001+00:00 (in 59.999509 seconds) quotaregistrysizeworker stdout | 2023-07-06 07:34:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:35:34 UTC)" (scheduled at 2023-07-06 07:34:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 07:34:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 07:34:34,895 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 07:34:34,895 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:35:34 UTC)" executed successfully quotatotalworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stderr | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stderr | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stderr | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stderr | self.send(msg) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stderr | self.connect() quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stderr | self.sock = self._create_connection( quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stderr | raise err quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stderr | sock.connect(sa) quotatotalworker stderr | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stderr | During handling of the above exception, another exception occurred: quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stderr | push_to_gateway( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stderr | handler( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stderr | response = self._open(req, data) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stderr | result = func(*args) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stderr | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stderr | raise URLError(err) quotatotalworker stderr | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:34:48,611 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stderr | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stderr | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stderr | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stderr | self.send(msg) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stderr | self.connect() quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stderr | self.sock = self._create_connection( quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stderr | raise err quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stderr | sock.connect(sa) quotatotalworker stderr | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stderr | During handling of the above exception, another exception occurred: quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stderr | push_to_gateway( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stderr | handler( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stderr | response = self._open(req, data) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stderr | result = func(*args) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stderr | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stderr | raise URLError(err) quotatotalworker stderr | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:35:18,644 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:35:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 07:35:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:36:34.887001+00:00 (in 59.999572 seconds) quotaregistrysizeworker stdout | 2023-07-06 07:35:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:36:34 UTC)" (scheduled at 2023-07-06 07:35:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 07:35:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 07:35:34,895 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 07:35:34,895 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:36:34 UTC)" executed successfully quotatotalworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stderr | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stderr | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stderr | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stderr | self.send(msg) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stderr | self.connect() quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stderr | self.sock = self._create_connection( quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stderr | raise err quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stderr | sock.connect(sa) quotatotalworker stderr | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stderr | During handling of the above exception, another exception occurred: quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stderr | push_to_gateway( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stderr | handler( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stderr | response = self._open(req, data) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stderr | result = func(*args) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stderr | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stderr | raise URLError(err) quotatotalworker stderr | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:35:48,676 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stderr | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stderr | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stderr | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stderr | self.send(msg) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stderr | self.connect() quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stderr | self.sock = self._create_connection( quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stderr | raise err quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stderr | sock.connect(sa) quotatotalworker stderr | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stderr | During handling of the above exception, another exception occurred: quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stderr | push_to_gateway( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stderr | handler( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stderr | response = self._open(req, data) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stderr | result = func(*args) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stderr | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stderr | raise URLError(err) quotatotalworker stderr | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:36:18,708 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:36:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 07:36:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:37:34.887001+00:00 (in 59.999633 seconds) quotaregistrysizeworker stdout | 2023-07-06 07:36:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:37:34 UTC)" (scheduled at 2023-07-06 07:36:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 07:36:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 07:36:34,895 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 07:36:34,895 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:37:34 UTC)" executed successfully quotatotalworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stderr | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stderr | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stderr | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stderr | self.send(msg) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stderr | self.connect() quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stderr | self.sock = self._create_connection( quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stderr | raise err quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stderr | sock.connect(sa) quotatotalworker stderr | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stderr | During handling of the above exception, another exception occurred: quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stderr | push_to_gateway( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stderr | handler( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stderr | response = self._open(req, data) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stderr | result = func(*args) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stderr | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stderr | raise URLError(err) quotatotalworker stderr | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:36:48,731 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stderr | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stderr | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stderr | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stderr | self.send(msg) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stderr | self.connect() quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stderr | self.sock = self._create_connection( quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stderr | raise err quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stderr | sock.connect(sa) quotatotalworker stderr | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stderr | During handling of the above exception, another exception occurred: quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stderr | push_to_gateway( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stderr | handler( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stderr | response = self._open(req, data) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stderr | result = func(*args) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stderr | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stderr | raise URLError(err) quotatotalworker stderr | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:37:18,747 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:37:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 07:37:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:38:34.887001+00:00 (in 59.999599 seconds) quotaregistrysizeworker stdout | 2023-07-06 07:37:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:38:34 UTC)" (scheduled at 2023-07-06 07:37:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 07:37:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 07:37:34,895 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 07:37:34,895 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:38:34 UTC)" executed successfully quotatotalworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stderr | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stderr | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stderr | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stderr | self.send(msg) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stderr | self.connect() quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stderr | self.sock = self._create_connection( quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stderr | raise err quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stderr | sock.connect(sa) quotatotalworker stderr | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stderr | During handling of the above exception, another exception occurred: quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stderr | push_to_gateway( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stderr | handler( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stderr | response = self._open(req, data) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stderr | result = func(*args) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stderr | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stderr | raise URLError(err) quotatotalworker stderr | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:37:48,779 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stderr | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stderr | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stderr | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stderr | self.send(msg) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stderr | self.connect() quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stderr | self.sock = self._create_connection( quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stderr | raise err quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stderr | sock.connect(sa) quotatotalworker stderr | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stderr | During handling of the above exception, another exception occurred: quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stderr | push_to_gateway( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stderr | handler( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stderr | response = self._open(req, data) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stderr | result = func(*args) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stderr | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stderr | raise URLError(err) quotatotalworker stderr | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:38:18,792 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:38:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 07:38:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:39:34.887001+00:00 (in 59.999585 seconds) quotaregistrysizeworker stdout | 2023-07-06 07:38:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:39:34 UTC)" (scheduled at 2023-07-06 07:38:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 07:38:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 07:38:34,895 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 07:38:34,895 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:39:34 UTC)" executed successfully quotatotalworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stderr | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stderr | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stderr | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stderr | self.send(msg) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stderr | self.connect() quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stderr | self.sock = self._create_connection( quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stderr | raise err quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stderr | sock.connect(sa) quotatotalworker stderr | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stderr | During handling of the above exception, another exception occurred: quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stderr | push_to_gateway( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stderr | handler( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stderr | response = self._open(req, data) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stderr | result = func(*args) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stderr | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stderr | raise URLError(err) quotatotalworker stderr | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:38:48,796 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stderr | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stderr | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stderr | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stderr | self.send(msg) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stderr | self.connect() quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stderr | self.sock = self._create_connection( quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stderr | raise err quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stderr | sock.connect(sa) quotatotalworker stderr | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stderr | During handling of the above exception, another exception occurred: quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stderr | push_to_gateway( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stderr | handler( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stderr | response = self._open(req, data) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stderr | result = func(*args) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stderr | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stderr | raise URLError(err) quotatotalworker stderr | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:39:18,801 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:39:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 07:39:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:40:34.887001+00:00 (in 59.999616 seconds) quotaregistrysizeworker stdout | 2023-07-06 07:39:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:40:34 UTC)" (scheduled at 2023-07-06 07:39:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 07:39:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 07:39:34,895 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 07:39:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:40:34 UTC)" executed successfully quotatotalworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stderr | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stderr | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stderr | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stderr | self.send(msg) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stderr | self.connect() quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stderr | self.sock = self._create_connection( quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stderr | raise err quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stderr | sock.connect(sa) quotatotalworker stderr | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stderr | During handling of the above exception, another exception occurred: quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stderr | push_to_gateway( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stderr | handler( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stderr | response = self._open(req, data) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stderr | result = func(*args) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stderr | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stderr | raise URLError(err) quotatotalworker stderr | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:39:48,807 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stderr | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stderr | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stderr | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stderr | self.send(msg) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stderr | self.connect() quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stderr | self.sock = self._create_connection( quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stderr | raise err quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stderr | sock.connect(sa) quotatotalworker stderr | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stderr | During handling of the above exception, another exception occurred: quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stderr | push_to_gateway( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stderr | handler( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stderr | response = self._open(req, data) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stderr | result = func(*args) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stderr | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stderr | raise URLError(err) quotatotalworker stderr | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:40:18,839 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:40:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 07:40:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:41:34.887001+00:00 (in 59.999603 seconds) quotaregistrysizeworker stdout | 2023-07-06 07:40:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:41:34 UTC)" (scheduled at 2023-07-06 07:40:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 07:40:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 07:40:34,895 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 07:40:34,895 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:41:34 UTC)" executed successfully quotatotalworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stderr | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stderr | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stderr | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stderr | self.send(msg) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stderr | self.connect() quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stderr | self.sock = self._create_connection( quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stderr | raise err quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stderr | sock.connect(sa) quotatotalworker stderr | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stderr | During handling of the above exception, another exception occurred: quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stderr | push_to_gateway( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stderr | handler( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stderr | response = self._open(req, data) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stderr | result = func(*args) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stderr | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stderr | raise URLError(err) quotatotalworker stderr | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:40:48,869 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stderr | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stderr | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stderr | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stderr | self.send(msg) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stderr | self.connect() quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stderr | self.sock = self._create_connection( quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stderr | raise err quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stderr | sock.connect(sa) quotatotalworker stderr | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stderr | During handling of the above exception, another exception occurred: quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stderr | push_to_gateway( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stderr | handler( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stderr | response = self._open(req, data) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stderr | result = func(*args) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stderr | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stderr | raise URLError(err) quotatotalworker stderr | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:41:18,891 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:41:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 07:41:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:42:34.887001+00:00 (in 59.999484 seconds) quotaregistrysizeworker stdout | 2023-07-06 07:41:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:42:34 UTC)" (scheduled at 2023-07-06 07:41:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 07:41:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 07:41:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 07:41:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:42:34 UTC)" executed successfully quotatotalworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stderr | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stderr | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stderr | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stderr | self.send(msg) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stderr | self.connect() quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stderr | self.sock = self._create_connection( quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stderr | raise err quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stderr | sock.connect(sa) quotatotalworker stderr | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stderr | During handling of the above exception, another exception occurred: quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stderr | push_to_gateway( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stderr | handler( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stderr | response = self._open(req, data) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stderr | result = func(*args) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stderr | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stderr | raise URLError(err) quotatotalworker stderr | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:41:48,907 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stderr | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stderr | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stderr | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stderr | self.send(msg) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stderr | self.connect() quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stderr | self.sock = self._create_connection( quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stderr | raise err quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stderr | sock.connect(sa) quotatotalworker stderr | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stderr | During handling of the above exception, another exception occurred: quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stderr | push_to_gateway( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stderr | handler( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stderr | response = self._open(req, data) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stderr | result = func(*args) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stderr | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stderr | raise URLError(err) quotatotalworker stderr | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:42:18,939 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:42:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 07:42:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:43:34.887001+00:00 (in 59.999482 seconds) quotaregistrysizeworker stdout | 2023-07-06 07:42:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:43:34 UTC)" (scheduled at 2023-07-06 07:42:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 07:42:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 07:42:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 07:42:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:43:34 UTC)" executed successfully quotatotalworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stderr | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stderr | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stderr | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stderr | self.send(msg) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stderr | self.connect() quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stderr | self.sock = self._create_connection( quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stderr | raise err quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stderr | sock.connect(sa) quotatotalworker stderr | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stderr | During handling of the above exception, another exception occurred: quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stderr | push_to_gateway( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stderr | handler( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stderr | response = self._open(req, data) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stderr | result = func(*args) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stderr | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stderr | raise URLError(err) quotatotalworker stderr | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:42:48,971 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stderr | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stderr | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stderr | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stderr | self.send(msg) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stderr | self.connect() quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stderr | self.sock = self._create_connection( quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stderr | raise err quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stderr | sock.connect(sa) quotatotalworker stderr | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stderr | During handling of the above exception, another exception occurred: quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stderr | push_to_gateway( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stderr | handler( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stderr | response = self._open(req, data) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stderr | result = func(*args) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stderr | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stderr | raise URLError(err) quotatotalworker stderr | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:43:18,999 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:43:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 07:43:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:44:34.887001+00:00 (in 59.999500 seconds) quotaregistrysizeworker stdout | 2023-07-06 07:43:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:44:34 UTC)" (scheduled at 2023-07-06 07:43:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 07:43:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 07:43:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 07:43:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:44:34 UTC)" executed successfully quotatotalworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stderr | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stderr | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stderr | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stderr | self.send(msg) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stderr | self.connect() quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stderr | self.sock = self._create_connection( quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stderr | raise err quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stderr | sock.connect(sa) quotatotalworker stderr | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stderr | During handling of the above exception, another exception occurred: quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stderr | push_to_gateway( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stderr | handler( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stderr | response = self._open(req, data) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stderr | result = func(*args) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stderr | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stderr | raise URLError(err) quotatotalworker stderr | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:43:49,031 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stderr | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stderr | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stderr | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stderr | self.send(msg) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stderr | self.connect() quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stderr | self.sock = self._create_connection( quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stderr | raise err quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stderr | sock.connect(sa) quotatotalworker stderr | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stderr | During handling of the above exception, another exception occurred: quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stderr | push_to_gateway( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stderr | handler( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stderr | response = self._open(req, data) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stderr | result = func(*args) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stderr | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stderr | raise URLError(err) quotatotalworker stderr | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:44:19,036 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:44:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 07:44:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:45:34.887001+00:00 (in 59.999503 seconds) quotaregistrysizeworker stdout | 2023-07-06 07:44:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:45:34 UTC)" (scheduled at 2023-07-06 07:44:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 07:44:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 07:44:34,900 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 07:44:34,900 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:45:34 UTC)" executed successfully quotatotalworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stderr | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stderr | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stderr | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stderr | self.send(msg) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stderr | self.connect() quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stderr | self.sock = self._create_connection( quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stderr | raise err quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stderr | sock.connect(sa) quotatotalworker stderr | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stderr | During handling of the above exception, another exception occurred: quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stderr | push_to_gateway( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stderr | handler( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stderr | response = self._open(req, data) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stderr | result = func(*args) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stderr | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stderr | raise URLError(err) quotatotalworker stderr | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:44:49,069 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stderr | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stderr | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stderr | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stderr | self.send(msg) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stderr | self.connect() quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stderr | self.sock = self._create_connection( quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stderr | raise err quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stderr | sock.connect(sa) quotatotalworker stderr | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stderr | During handling of the above exception, another exception occurred: quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stderr | push_to_gateway( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stderr | handler( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stderr | response = self._open(req, data) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stderr | result = func(*args) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stderr | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stderr | raise URLError(err) quotatotalworker stderr | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:45:19,078 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:45:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 07:45:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:46:34.887001+00:00 (in 59.999479 seconds) quotaregistrysizeworker stdout | 2023-07-06 07:45:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:46:34 UTC)" (scheduled at 2023-07-06 07:45:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 07:45:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 07:45:34,895 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 07:45:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:46:34 UTC)" executed successfully quotatotalworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stderr | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stderr | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stderr | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stderr | self.send(msg) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stderr | self.connect() quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stderr | self.sock = self._create_connection( quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stderr | raise err quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stderr | sock.connect(sa) quotatotalworker stderr | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stderr | During handling of the above exception, another exception occurred: quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stderr | push_to_gateway( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stderr | handler( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stderr | response = self._open(req, data) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stderr | result = func(*args) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stderr | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stderr | raise URLError(err) quotatotalworker stderr | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:45:49,096 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stderr | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stderr | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stderr | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stderr | self.send(msg) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stderr | self.connect() quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stderr | self.sock = self._create_connection( quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stderr | raise err quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stderr | sock.connect(sa) quotatotalworker stderr | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stderr | During handling of the above exception, another exception occurred: quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stderr | push_to_gateway( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stderr | handler( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stderr | response = self._open(req, data) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stderr | result = func(*args) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stderr | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stderr | raise URLError(err) quotatotalworker stderr | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:46:19,124 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:46:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 07:46:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:47:34.887001+00:00 (in 59.999499 seconds) quotaregistrysizeworker stdout | 2023-07-06 07:46:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:47:34 UTC)" (scheduled at 2023-07-06 07:46:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 07:46:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 07:46:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 07:46:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:47:34 UTC)" executed successfully quotatotalworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stderr | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stderr | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stderr | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stderr | self.send(msg) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stderr | self.connect() quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stderr | self.sock = self._create_connection( quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stderr | raise err quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stderr | sock.connect(sa) quotatotalworker stderr | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stderr | During handling of the above exception, another exception occurred: quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stderr | push_to_gateway( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stderr | handler( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stderr | response = self._open(req, data) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stderr | result = func(*args) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stderr | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stderr | raise URLError(err) quotatotalworker stderr | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:46:49,156 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stderr | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stderr | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stderr | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stderr | self.send(msg) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stderr | self.connect() quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stderr | self.sock = self._create_connection( quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stderr | raise err quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stderr | sock.connect(sa) quotatotalworker stderr | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stderr | During handling of the above exception, another exception occurred: quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stderr | push_to_gateway( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stderr | handler( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stderr | response = self._open(req, data) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stderr | result = func(*args) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stderr | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stderr | raise URLError(err) quotatotalworker stderr | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:47:19,179 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:47:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 07:47:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:48:34.887001+00:00 (in 59.999535 seconds) quotaregistrysizeworker stdout | 2023-07-06 07:47:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:48:34 UTC)" (scheduled at 2023-07-06 07:47:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 07:47:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 07:47:34,895 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 07:47:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:48:34 UTC)" executed successfully quotatotalworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stderr | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stderr | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stderr | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stderr | self.send(msg) quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stderr | self.connect() quotatotalworker stderr | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stderr | self.sock = self._create_connection( quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stderr | raise err quotatotalworker stderr | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stderr | sock.connect(sa) quotatotalworker stderr | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stderr | During handling of the above exception, another exception occurred: quotatotalworker stderr | Traceback (most recent call last): quotatotalworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stderr | push_to_gateway( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stderr | handler( quotatotalworker stderr | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stderr | response = self._open(req, data) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stderr | result = func(*args) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stderr | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stderr | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stderr | raise URLError(err) quotatotalworker stderr | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:47:49,197 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 07:48:17,977 [33] [DEBUG] [workers.worker] Scheduling worker. quotatotalworker stdout | 2023-07-06 07:48:17,978 [33] [INFO] [apscheduler.scheduler] Scheduler started quotatotalworker stdout | 2023-07-06 07:48:17,978 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 07:48:17,978 [33] [DEBUG] [workers.worker] First run scheduled for 2023-07-06 07:48:21.979497 quotatotalworker stdout | 2023-07-06 07:48:17,978 [33] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added quotatotalworker stdout | 2023-07-06 07:48:18,025 [33] [INFO] [apscheduler.scheduler] Added job "QuotaTotalWorker.backfill" to job store "default" quotatotalworker stdout | 2023-07-06 07:48:18,026 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 07:48:18,026 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:48:21.979497+00:00 (in 3.953373 seconds) quotatotalworker stdout | 2023-07-06 07:48:19,115 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:48:19,228 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 07:48:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 07:48:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:48:21 UTC)" (scheduled at 2023-07-06 07:48:21.979497+00:00) quotatotalworker stdout | 2023-07-06 07:48:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 07:48:21,982 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:48:36.979497+00:00 (in 14.999725 seconds) quotatotalworker stdout | 2023-07-06 07:48:21,989 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."size_bytes", "t1"."backfill_start_ms", "t1"."backfill_complete" FROM "quotanamespacesize" AS "t1" WHERE ("t1"."namespace_user_id" = %s) LIMIT %s OFFSET %s', [5, 1, 0]) quotatotalworker stdout | 2023-07-06 07:48:21,991 [33] [DEBUG] [peewee] ('INSERT INTO "quotanamespacesize" ("namespace_user_id", "size_bytes", "backfill_start_ms", "backfill_complete") VALUES (%s, %s, %s, %s) RETURNING "quotanamespacesize"."id"', [5, 0, 1688629701991, False]) quotatotalworker stdout | 2023-07-06 07:48:21,995 [33] [DEBUG] [peewee] ('SELECT Sum("t1"."image_size") FROM (SELECT "t2"."image_size" FROM "imagestorage" AS "t2" INNER JOIN "manifestblob" AS "t3" ON ("t2"."id" = "t3"."blob_id") INNER JOIN "repository" AS "t4" ON ("t4"."id" = "t3"."repository_id") WHERE ("t4"."namespace_user_id" = %s) GROUP BY "t2"."id") AS "t1"', [5]) quotatotalworker stdout | 2023-07-06 07:48:21,998 [33] [DEBUG] [peewee] ('UPDATE "quotanamespacesize" SET "size_bytes" = %s, "backfill_complete" = %s WHERE ("quotanamespacesize"."namespace_user_id" = %s)', [0, True, 5]) quotatotalworker stdout | 2023-07-06 07:48:22,012 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state" FROM "repository" AS "t1" WHERE (("t1"."namespace_user_id" = %s) AND ("t1"."state" != %s))', [5, 3]) quotatotalworker stdout | 2023-07-06 07:48:22,014 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."size_bytes", "t1"."backfill_start_ms", "t1"."backfill_complete" FROM "quotanamespacesize" AS "t1" WHERE ("t1"."namespace_user_id" = %s) LIMIT %s OFFSET %s', [6, 1, 0]) quotatotalworker stdout | 2023-07-06 07:48:22,015 [33] [DEBUG] [peewee] ('INSERT INTO "quotanamespacesize" ("namespace_user_id", "size_bytes", "backfill_start_ms", "backfill_complete") VALUES (%s, %s, %s, %s) RETURNING "quotanamespacesize"."id"', [6, 0, 1688629702015, False]) quotatotalworker stdout | 2023-07-06 07:48:22,019 [33] [DEBUG] [peewee] ('SELECT Sum("t1"."image_size") FROM (SELECT "t2"."image_size" FROM "imagestorage" AS "t2" INNER JOIN "manifestblob" AS "t3" ON ("t2"."id" = "t3"."blob_id") INNER JOIN "repository" AS "t4" ON ("t4"."id" = "t3"."repository_id") WHERE ("t4"."namespace_user_id" = %s) GROUP BY "t2"."id") AS "t1"', [6]) quotatotalworker stdout | 2023-07-06 07:48:22,021 [33] [DEBUG] [peewee] ('UPDATE "quotanamespacesize" SET "size_bytes" = %s, "backfill_complete" = %s WHERE ("quotanamespacesize"."namespace_user_id" = %s)', [0, True, 6]) quotatotalworker stdout | 2023-07-06 07:48:22,024 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state" FROM "repository" AS "t1" WHERE (("t1"."namespace_user_id" = %s) AND ("t1"."state" != %s))', [6, 3]) quotatotalworker stdout | 2023-07-06 07:48:22,026 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 07:48:22,026 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:48:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 07:48:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 07:48:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:49:34.887001+00:00 (in 59.999516 seconds) quotaregistrysizeworker stdout | 2023-07-06 07:48:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:49:34 UTC)" (scheduled at 2023-07-06 07:48:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 07:48:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 07:48:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 07:48:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:49:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 07:48:36,981 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 07:48:36,982 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:48:51.979497+00:00 (in 14.997444 seconds) quotatotalworker stdout | 2023-07-06 07:48:36,982 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:48:51 UTC)" (scheduled at 2023-07-06 07:48:36.979497+00:00) quotatotalworker stdout | 2023-07-06 07:48:36,983 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 07:48:36,991 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 07:48:36,991 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:48:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 07:48:49,145 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:48:49,246 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 07:48:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 07:48:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:49:06.979497+00:00 (in 14.999576 seconds) quotatotalworker stdout | 2023-07-06 07:48:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:49:06 UTC)" (scheduled at 2023-07-06 07:48:51.979497+00:00) quotatotalworker stdout | 2023-07-06 07:48:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 07:48:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 07:48:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:49:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 07:49:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 07:49:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:49:21.979497+00:00 (in 14.999522 seconds) quotatotalworker stdout | 2023-07-06 07:49:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:49:21 UTC)" (scheduled at 2023-07-06 07:49:06.979497+00:00) quotatotalworker stdout | 2023-07-06 07:49:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 07:49:06,990 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 07:49:06,990 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:49:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 07:49:19,160 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:49:19,279 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 07:49:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 07:49:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:49:36.979497+00:00 (in 14.999534 seconds) quotatotalworker stdout | 2023-07-06 07:49:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:49:36 UTC)" (scheduled at 2023-07-06 07:49:21.979497+00:00) quotatotalworker stdout | 2023-07-06 07:49:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 07:49:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 07:49:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:49:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 07:49:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 07:49:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:50:34.887001+00:00 (in 59.999470 seconds) quotaregistrysizeworker stdout | 2023-07-06 07:49:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:50:34 UTC)" (scheduled at 2023-07-06 07:49:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 07:49:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 07:49:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 07:49:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:50:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 07:49:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 07:49:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:49:51.979497+00:00 (in 14.999546 seconds) quotatotalworker stdout | 2023-07-06 07:49:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:49:51 UTC)" (scheduled at 2023-07-06 07:49:36.979497+00:00) quotatotalworker stdout | 2023-07-06 07:49:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 07:49:36,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 07:49:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:49:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 07:49:49,170 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:49:49,281 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 07:49:51,980 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 07:49:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:50:06.979497+00:00 (in 14.998987 seconds) quotatotalworker stdout | 2023-07-06 07:49:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:50:06 UTC)" (scheduled at 2023-07-06 07:49:51.979497+00:00) quotatotalworker stdout | 2023-07-06 07:49:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 07:49:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 07:49:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:50:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 07:50:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 07:50:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:50:21.979497+00:00 (in 14.999561 seconds) quotatotalworker stdout | 2023-07-06 07:50:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:50:21 UTC)" (scheduled at 2023-07-06 07:50:06.979497+00:00) quotatotalworker stdout | 2023-07-06 07:50:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 07:50:06,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 07:50:06,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:50:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 07:50:19,175 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:50:19,283 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 07:50:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 07:50:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:50:36.979497+00:00 (in 14.999555 seconds) quotatotalworker stdout | 2023-07-06 07:50:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:50:36 UTC)" (scheduled at 2023-07-06 07:50:21.979497+00:00) quotatotalworker stdout | 2023-07-06 07:50:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 07:50:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 07:50:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:50:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 07:50:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 07:50:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:51:34.887001+00:00 (in 59.999509 seconds) quotaregistrysizeworker stdout | 2023-07-06 07:50:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:51:34 UTC)" (scheduled at 2023-07-06 07:50:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 07:50:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 07:50:34,895 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 07:50:34,895 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:51:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 07:50:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 07:50:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:50:51.979497+00:00 (in 14.999565 seconds) quotatotalworker stdout | 2023-07-06 07:50:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:50:51 UTC)" (scheduled at 2023-07-06 07:50:36.979497+00:00) quotatotalworker stdout | 2023-07-06 07:50:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 07:50:36,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 07:50:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:50:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 07:50:49,207 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:50:49,315 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 07:50:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 07:50:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:51:06.979497+00:00 (in 14.999581 seconds) quotatotalworker stdout | 2023-07-06 07:50:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:51:06 UTC)" (scheduled at 2023-07-06 07:50:51.979497+00:00) quotatotalworker stdout | 2023-07-06 07:50:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 07:50:51,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 07:50:51,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:51:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 07:51:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 07:51:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:51:21.979497+00:00 (in 14.999509 seconds) quotatotalworker stdout | 2023-07-06 07:51:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:51:21 UTC)" (scheduled at 2023-07-06 07:51:06.979497+00:00) quotatotalworker stdout | 2023-07-06 07:51:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 07:51:06,990 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 07:51:06,990 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:51:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 07:51:19,210 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:51:19,347 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 07:51:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 07:51:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:51:36.979497+00:00 (in 14.999448 seconds) quotatotalworker stdout | 2023-07-06 07:51:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:51:36 UTC)" (scheduled at 2023-07-06 07:51:21.979497+00:00) quotatotalworker stdout | 2023-07-06 07:51:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 07:51:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 07:51:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:51:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 07:51:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 07:51:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:52:34.887001+00:00 (in 59.999530 seconds) quotaregistrysizeworker stdout | 2023-07-06 07:51:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:52:34 UTC)" (scheduled at 2023-07-06 07:51:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 07:51:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 07:51:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 07:51:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:52:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 07:51:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 07:51:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:51:51.979497+00:00 (in 14.999517 seconds) quotatotalworker stdout | 2023-07-06 07:51:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:51:51 UTC)" (scheduled at 2023-07-06 07:51:36.979497+00:00) quotatotalworker stdout | 2023-07-06 07:51:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 07:51:36,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 07:51:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:51:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 07:51:49,213 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:51:49,379 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 07:51:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 07:51:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:52:06.979497+00:00 (in 14.999525 seconds) quotatotalworker stdout | 2023-07-06 07:51:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:52:06 UTC)" (scheduled at 2023-07-06 07:51:51.979497+00:00) quotatotalworker stdout | 2023-07-06 07:51:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 07:51:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 07:51:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:52:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 07:52:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 07:52:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:52:21.979497+00:00 (in 14.999532 seconds) quotatotalworker stdout | 2023-07-06 07:52:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:52:21 UTC)" (scheduled at 2023-07-06 07:52:06.979497+00:00) quotatotalworker stdout | 2023-07-06 07:52:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 07:52:06,990 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 07:52:06,990 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:52:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 07:52:19,245 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:52:19,411 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 07:52:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 07:52:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:52:36.979497+00:00 (in 14.999569 seconds) quotatotalworker stdout | 2023-07-06 07:52:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:52:36 UTC)" (scheduled at 2023-07-06 07:52:21.979497+00:00) quotatotalworker stdout | 2023-07-06 07:52:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 07:52:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 07:52:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:52:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 07:52:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 07:52:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:53:34.887001+00:00 (in 59.999540 seconds) quotaregistrysizeworker stdout | 2023-07-06 07:52:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:53:34 UTC)" (scheduled at 2023-07-06 07:52:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 07:52:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 07:52:34,895 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 07:52:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:53:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 07:52:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 07:52:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:52:51.979497+00:00 (in 14.999551 seconds) quotatotalworker stdout | 2023-07-06 07:52:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:52:51 UTC)" (scheduled at 2023-07-06 07:52:36.979497+00:00) quotatotalworker stdout | 2023-07-06 07:52:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 07:52:36,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 07:52:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:52:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 07:52:49,276 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:52:49,439 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 07:52:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 07:52:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:53:06.979497+00:00 (in 14.999539 seconds) quotatotalworker stdout | 2023-07-06 07:52:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:53:06 UTC)" (scheduled at 2023-07-06 07:52:51.979497+00:00) quotatotalworker stdout | 2023-07-06 07:52:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 07:52:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 07:52:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:53:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 07:53:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 07:53:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:53:21.979497+00:00 (in 14.999509 seconds) quotatotalworker stdout | 2023-07-06 07:53:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:53:21 UTC)" (scheduled at 2023-07-06 07:53:06.979497+00:00) quotatotalworker stdout | 2023-07-06 07:53:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 07:53:06,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 07:53:06,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:53:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 07:53:19,303 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:53:19,471 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 07:53:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 07:53:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:53:36.979497+00:00 (in 14.999533 seconds) quotatotalworker stdout | 2023-07-06 07:53:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:53:36 UTC)" (scheduled at 2023-07-06 07:53:21.979497+00:00) quotatotalworker stdout | 2023-07-06 07:53:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 07:53:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 07:53:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:53:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 07:53:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 07:53:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:54:34.887001+00:00 (in 59.999562 seconds) quotaregistrysizeworker stdout | 2023-07-06 07:53:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:54:34 UTC)" (scheduled at 2023-07-06 07:53:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 07:53:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 07:53:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 07:53:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:54:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 07:53:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 07:53:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:53:51.979497+00:00 (in 14.999502 seconds) quotatotalworker stdout | 2023-07-06 07:53:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:53:51 UTC)" (scheduled at 2023-07-06 07:53:36.979497+00:00) quotatotalworker stdout | 2023-07-06 07:53:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 07:53:36,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 07:53:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:53:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 07:53:49,330 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:53:49,498 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 07:53:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 07:53:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:54:06.979497+00:00 (in 14.999536 seconds) quotatotalworker stdout | 2023-07-06 07:53:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:54:06 UTC)" (scheduled at 2023-07-06 07:53:51.979497+00:00) quotatotalworker stdout | 2023-07-06 07:53:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 07:53:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 07:53:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:54:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 07:54:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 07:54:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:54:21.979497+00:00 (in 14.999529 seconds) quotatotalworker stdout | 2023-07-06 07:54:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:54:21 UTC)" (scheduled at 2023-07-06 07:54:06.979497+00:00) quotatotalworker stdout | 2023-07-06 07:54:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 07:54:06,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 07:54:06,990 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:54:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 07:54:19,362 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:54:19,513 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 07:54:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 07:54:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:54:36.979497+00:00 (in 14.999554 seconds) quotatotalworker stdout | 2023-07-06 07:54:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:54:36 UTC)" (scheduled at 2023-07-06 07:54:21.979497+00:00) quotatotalworker stdout | 2023-07-06 07:54:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 07:54:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 07:54:21,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:54:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 07:54:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 07:54:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:55:34.887001+00:00 (in 59.999539 seconds) quotaregistrysizeworker stdout | 2023-07-06 07:54:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:55:34 UTC)" (scheduled at 2023-07-06 07:54:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 07:54:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 07:54:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 07:54:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:55:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 07:54:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 07:54:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:54:51.979497+00:00 (in 14.999501 seconds) quotatotalworker stdout | 2023-07-06 07:54:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:54:51 UTC)" (scheduled at 2023-07-06 07:54:36.979497+00:00) quotatotalworker stdout | 2023-07-06 07:54:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 07:54:36,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 07:54:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:54:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 07:54:49,391 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:54:49,538 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 07:54:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 07:54:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:55:06.979497+00:00 (in 14.999539 seconds) quotatotalworker stdout | 2023-07-06 07:54:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:55:06 UTC)" (scheduled at 2023-07-06 07:54:51.979497+00:00) quotatotalworker stdout | 2023-07-06 07:54:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 07:54:51,987 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 07:54:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:55:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 07:55:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 07:55:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:55:21.979497+00:00 (in 14.999492 seconds) quotatotalworker stdout | 2023-07-06 07:55:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:55:21 UTC)" (scheduled at 2023-07-06 07:55:06.979497+00:00) quotatotalworker stdout | 2023-07-06 07:55:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 07:55:06,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 07:55:06,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:55:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 07:55:19,423 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:55:19,570 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 07:55:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 07:55:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:55:36.979497+00:00 (in 14.999493 seconds) quotatotalworker stdout | 2023-07-06 07:55:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:55:36 UTC)" (scheduled at 2023-07-06 07:55:21.979497+00:00) quotatotalworker stdout | 2023-07-06 07:55:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 07:55:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 07:55:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:55:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 07:55:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 07:55:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:56:34.887001+00:00 (in 59.999562 seconds) quotaregistrysizeworker stdout | 2023-07-06 07:55:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:56:34 UTC)" (scheduled at 2023-07-06 07:55:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 07:55:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 07:55:34,895 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 07:55:34,895 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:56:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 07:55:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 07:55:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:55:51.979497+00:00 (in 14.999521 seconds) quotatotalworker stdout | 2023-07-06 07:55:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:55:51 UTC)" (scheduled at 2023-07-06 07:55:36.979497+00:00) quotatotalworker stdout | 2023-07-06 07:55:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 07:55:36,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 07:55:36,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:55:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 07:55:49,433 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:55:49,584 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 07:55:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 07:55:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:56:06.979497+00:00 (in 14.999540 seconds) quotatotalworker stdout | 2023-07-06 07:55:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:56:06 UTC)" (scheduled at 2023-07-06 07:55:51.979497+00:00) quotatotalworker stdout | 2023-07-06 07:55:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 07:55:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 07:55:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:56:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 07:56:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 07:56:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:56:21.979497+00:00 (in 14.999558 seconds) quotatotalworker stdout | 2023-07-06 07:56:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:56:21 UTC)" (scheduled at 2023-07-06 07:56:06.979497+00:00) quotatotalworker stdout | 2023-07-06 07:56:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 07:56:06,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 07:56:06,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:56:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 07:56:19,439 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:56:19,614 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 07:56:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 07:56:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:56:36.979497+00:00 (in 14.999563 seconds) quotatotalworker stdout | 2023-07-06 07:56:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:56:36 UTC)" (scheduled at 2023-07-06 07:56:21.979497+00:00) quotatotalworker stdout | 2023-07-06 07:56:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 07:56:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 07:56:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:56:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 07:56:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 07:56:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:57:34.887001+00:00 (in 59.999544 seconds) quotaregistrysizeworker stdout | 2023-07-06 07:56:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:57:34 UTC)" (scheduled at 2023-07-06 07:56:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 07:56:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 07:56:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 07:56:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:57:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 07:56:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 07:56:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:56:51.979497+00:00 (in 14.999520 seconds) quotatotalworker stdout | 2023-07-06 07:56:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:56:51 UTC)" (scheduled at 2023-07-06 07:56:36.979497+00:00) quotatotalworker stdout | 2023-07-06 07:56:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 07:56:36,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 07:56:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:56:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 07:56:49,452 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:56:49,620 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 07:56:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 07:56:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:57:06.979497+00:00 (in 14.999556 seconds) quotatotalworker stdout | 2023-07-06 07:56:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:57:06 UTC)" (scheduled at 2023-07-06 07:56:51.979497+00:00) quotatotalworker stdout | 2023-07-06 07:56:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 07:56:51,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 07:56:51,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:57:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 07:57:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 07:57:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:57:21.979497+00:00 (in 14.999513 seconds) quotatotalworker stdout | 2023-07-06 07:57:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:57:21 UTC)" (scheduled at 2023-07-06 07:57:06.979497+00:00) quotatotalworker stdout | 2023-07-06 07:57:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 07:57:06,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 07:57:06,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:57:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 07:57:19,458 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:57:19,652 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 07:57:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 07:57:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:57:36.979497+00:00 (in 14.999541 seconds) quotatotalworker stdout | 2023-07-06 07:57:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:57:36 UTC)" (scheduled at 2023-07-06 07:57:21.979497+00:00) quotatotalworker stdout | 2023-07-06 07:57:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 07:57:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 07:57:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:57:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 07:57:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 07:57:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:58:34.887001+00:00 (in 59.999487 seconds) quotaregistrysizeworker stdout | 2023-07-06 07:57:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:58:34 UTC)" (scheduled at 2023-07-06 07:57:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 07:57:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 07:57:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 07:57:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:58:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 07:57:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 07:57:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:57:51.979497+00:00 (in 14.999545 seconds) quotatotalworker stdout | 2023-07-06 07:57:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:57:51 UTC)" (scheduled at 2023-07-06 07:57:36.979497+00:00) quotatotalworker stdout | 2023-07-06 07:57:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 07:57:36,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 07:57:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:57:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 07:57:49,484 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:57:49,684 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 07:57:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 07:57:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:58:06.979497+00:00 (in 14.999541 seconds) quotatotalworker stdout | 2023-07-06 07:57:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:58:06 UTC)" (scheduled at 2023-07-06 07:57:51.979497+00:00) quotatotalworker stdout | 2023-07-06 07:57:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 07:57:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 07:57:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:58:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 07:58:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 07:58:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:58:21.979497+00:00 (in 14.999534 seconds) quotatotalworker stdout | 2023-07-06 07:58:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:58:21 UTC)" (scheduled at 2023-07-06 07:58:06.979497+00:00) quotatotalworker stdout | 2023-07-06 07:58:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 07:58:06,990 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 07:58:06,990 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:58:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 07:58:19,507 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:58:19,700 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 07:58:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 07:58:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:58:36.979497+00:00 (in 14.999532 seconds) quotatotalworker stdout | 2023-07-06 07:58:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:58:36 UTC)" (scheduled at 2023-07-06 07:58:21.979497+00:00) quotatotalworker stdout | 2023-07-06 07:58:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 07:58:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 07:58:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:58:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 07:58:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 07:58:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:59:34.887001+00:00 (in 59.999517 seconds) quotaregistrysizeworker stdout | 2023-07-06 07:58:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:59:34 UTC)" (scheduled at 2023-07-06 07:58:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 07:58:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 07:58:34,906 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 07:58:34,906 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 07:59:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 07:58:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 07:58:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:58:51.979497+00:00 (in 14.999535 seconds) quotatotalworker stdout | 2023-07-06 07:58:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:58:51 UTC)" (scheduled at 2023-07-06 07:58:36.979497+00:00) quotatotalworker stdout | 2023-07-06 07:58:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 07:58:36,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 07:58:36,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:58:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 07:58:49,539 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:58:49,731 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 07:58:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 07:58:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:59:06.979497+00:00 (in 14.999475 seconds) quotatotalworker stdout | 2023-07-06 07:58:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:59:06 UTC)" (scheduled at 2023-07-06 07:58:51.979497+00:00) quotatotalworker stdout | 2023-07-06 07:58:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 07:58:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 07:58:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:59:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 07:59:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 07:59:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:59:21.979497+00:00 (in 14.999497 seconds) quotatotalworker stdout | 2023-07-06 07:59:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:59:21 UTC)" (scheduled at 2023-07-06 07:59:06.979497+00:00) quotatotalworker stdout | 2023-07-06 07:59:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 07:59:06,990 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 07:59:06,990 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:59:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 07:59:19,571 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:59:19,763 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 07:59:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 07:59:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:59:36.979497+00:00 (in 14.999474 seconds) quotatotalworker stdout | 2023-07-06 07:59:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:59:36 UTC)" (scheduled at 2023-07-06 07:59:21.979497+00:00) quotatotalworker stdout | 2023-07-06 07:59:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 07:59:21,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 07:59:21,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:59:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 07:59:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 07:59:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:00:34.887001+00:00 (in 59.999545 seconds) quotaregistrysizeworker stdout | 2023-07-06 07:59:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:00:34 UTC)" (scheduled at 2023-07-06 07:59:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 07:59:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 07:59:34,895 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 07:59:34,895 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:00:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 07:59:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 07:59:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 07:59:51.979497+00:00 (in 14.999473 seconds) quotatotalworker stdout | 2023-07-06 07:59:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:59:51 UTC)" (scheduled at 2023-07-06 07:59:36.979497+00:00) quotatotalworker stdout | 2023-07-06 07:59:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 07:59:36,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 07:59:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 07:59:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 07:59:49,578 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 07:59:49,795 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 07:59:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 07:59:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:00:06.979497+00:00 (in 14.999535 seconds) quotatotalworker stdout | 2023-07-06 07:59:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:00:06 UTC)" (scheduled at 2023-07-06 07:59:51.979497+00:00) quotatotalworker stdout | 2023-07-06 07:59:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 07:59:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 07:59:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:00:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:00:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:00:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:00:21.979497+00:00 (in 14.999532 seconds) quotatotalworker stdout | 2023-07-06 08:00:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:00:21 UTC)" (scheduled at 2023-07-06 08:00:06.979497+00:00) quotatotalworker stdout | 2023-07-06 08:00:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:00:06,990 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:00:06,990 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:00:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:00:19,594 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:00:19,828 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:00:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:00:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:00:36.979497+00:00 (in 14.999598 seconds) quotatotalworker stdout | 2023-07-06 08:00:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:00:36 UTC)" (scheduled at 2023-07-06 08:00:21.979497+00:00) quotatotalworker stdout | 2023-07-06 08:00:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:00:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:00:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:00:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 08:00:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 08:00:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:01:34.887001+00:00 (in 59.999506 seconds) quotaregistrysizeworker stdout | 2023-07-06 08:00:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:01:34 UTC)" (scheduled at 2023-07-06 08:00:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 08:00:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 08:00:34,895 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 08:00:34,895 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:01:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:00:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:00:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:00:51.979497+00:00 (in 14.999544 seconds) quotatotalworker stdout | 2023-07-06 08:00:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:00:51 UTC)" (scheduled at 2023-07-06 08:00:36.979497+00:00) quotatotalworker stdout | 2023-07-06 08:00:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:00:36,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:00:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:00:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:00:49,625 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:00:49,860 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:00:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:00:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:01:06.979497+00:00 (in 14.999569 seconds) quotatotalworker stdout | 2023-07-06 08:00:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:01:06 UTC)" (scheduled at 2023-07-06 08:00:51.979497+00:00) quotatotalworker stdout | 2023-07-06 08:00:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:00:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:00:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:01:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:01:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:01:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:01:21.979497+00:00 (in 14.999561 seconds) quotatotalworker stdout | 2023-07-06 08:01:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:01:21 UTC)" (scheduled at 2023-07-06 08:01:06.979497+00:00) quotatotalworker stdout | 2023-07-06 08:01:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:01:06,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:01:06,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:01:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:01:19,657 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:01:19,892 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:01:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:01:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:01:36.979497+00:00 (in 14.999596 seconds) quotatotalworker stdout | 2023-07-06 08:01:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:01:36 UTC)" (scheduled at 2023-07-06 08:01:21.979497+00:00) quotatotalworker stdout | 2023-07-06 08:01:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:01:21,990 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:01:21,990 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:01:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 08:01:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 08:01:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:02:34.887001+00:00 (in 59.999476 seconds) quotaregistrysizeworker stdout | 2023-07-06 08:01:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:02:34 UTC)" (scheduled at 2023-07-06 08:01:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 08:01:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 08:01:34,895 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 08:01:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:02:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:01:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:01:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:01:51.979497+00:00 (in 14.999538 seconds) quotatotalworker stdout | 2023-07-06 08:01:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:01:51 UTC)" (scheduled at 2023-07-06 08:01:36.979497+00:00) quotatotalworker stdout | 2023-07-06 08:01:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:01:36,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:01:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:01:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:01:49,690 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:01:49,924 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:01:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:01:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:02:06.979497+00:00 (in 14.999560 seconds) quotatotalworker stdout | 2023-07-06 08:01:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:02:06 UTC)" (scheduled at 2023-07-06 08:01:51.979497+00:00) quotatotalworker stdout | 2023-07-06 08:01:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:01:51,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:01:51,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:02:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:02:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:02:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:02:21.979497+00:00 (in 14.999564 seconds) quotatotalworker stdout | 2023-07-06 08:02:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:02:21 UTC)" (scheduled at 2023-07-06 08:02:06.979497+00:00) quotatotalworker stdout | 2023-07-06 08:02:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:02:06,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:02:06,990 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:02:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:02:19,722 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:02:19,956 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:02:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:02:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:02:36.979497+00:00 (in 14.999582 seconds) quotatotalworker stdout | 2023-07-06 08:02:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:02:36 UTC)" (scheduled at 2023-07-06 08:02:21.979497+00:00) quotatotalworker stdout | 2023-07-06 08:02:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:02:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:02:21,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:02:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 08:02:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 08:02:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:03:34.887001+00:00 (in 59.999502 seconds) quotaregistrysizeworker stdout | 2023-07-06 08:02:34,888 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:03:34 UTC)" (scheduled at 2023-07-06 08:02:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 08:02:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 08:02:34,897 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 08:02:34,897 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:03:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:02:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:02:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:02:51.979497+00:00 (in 14.999499 seconds) quotatotalworker stdout | 2023-07-06 08:02:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:02:51 UTC)" (scheduled at 2023-07-06 08:02:36.979497+00:00) quotatotalworker stdout | 2023-07-06 08:02:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:02:36,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:02:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:02:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:02:49,754 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:02:49,989 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:02:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:02:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:03:06.979497+00:00 (in 14.999513 seconds) quotatotalworker stdout | 2023-07-06 08:02:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:03:06 UTC)" (scheduled at 2023-07-06 08:02:51.979497+00:00) quotatotalworker stdout | 2023-07-06 08:02:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:02:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:02:51,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:03:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:03:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:03:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:03:21.979497+00:00 (in 14.999543 seconds) quotatotalworker stdout | 2023-07-06 08:03:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:03:21 UTC)" (scheduled at 2023-07-06 08:03:06.979497+00:00) quotatotalworker stdout | 2023-07-06 08:03:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:03:06,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:03:06,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:03:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:03:19,781 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:03:20,021 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:03:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:03:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:03:36.979497+00:00 (in 14.999574 seconds) quotatotalworker stdout | 2023-07-06 08:03:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:03:36 UTC)" (scheduled at 2023-07-06 08:03:21.979497+00:00) quotatotalworker stdout | 2023-07-06 08:03:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:03:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:03:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:03:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 08:03:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 08:03:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:04:34.887001+00:00 (in 59.999507 seconds) quotaregistrysizeworker stdout | 2023-07-06 08:03:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:04:34 UTC)" (scheduled at 2023-07-06 08:03:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 08:03:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 08:03:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 08:03:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:04:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:03:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:03:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:03:51.979497+00:00 (in 14.999579 seconds) quotatotalworker stdout | 2023-07-06 08:03:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:03:51 UTC)" (scheduled at 2023-07-06 08:03:36.979497+00:00) quotatotalworker stdout | 2023-07-06 08:03:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:03:36,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:03:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:03:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:03:49,796 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:03:50,034 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:03:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:03:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:04:06.979497+00:00 (in 14.999573 seconds) quotatotalworker stdout | 2023-07-06 08:03:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:04:06 UTC)" (scheduled at 2023-07-06 08:03:51.979497+00:00) quotatotalworker stdout | 2023-07-06 08:03:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:03:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:03:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:04:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:04:06,980 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:04:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:04:21.979497+00:00 (in 14.999048 seconds) quotatotalworker stdout | 2023-07-06 08:04:07,008 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:04:21 UTC)" (scheduled at 2023-07-06 08:04:06.979497+00:00) quotatotalworker stdout | 2023-07-06 08:04:07,009 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:04:07,017 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:04:07,017 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:04:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:04:19,828 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:04:20,058 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:04:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:04:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:04:36.979497+00:00 (in 14.999457 seconds) quotatotalworker stdout | 2023-07-06 08:04:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:04:36 UTC)" (scheduled at 2023-07-06 08:04:21.979497+00:00) quotatotalworker stdout | 2023-07-06 08:04:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:04:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:04:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:04:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 08:04:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 08:04:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:05:34.887001+00:00 (in 59.999442 seconds) quotaregistrysizeworker stdout | 2023-07-06 08:04:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:05:34 UTC)" (scheduled at 2023-07-06 08:04:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 08:04:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 08:04:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 08:04:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:05:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:04:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:04:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:04:51.979497+00:00 (in 14.999487 seconds) quotatotalworker stdout | 2023-07-06 08:04:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:04:51 UTC)" (scheduled at 2023-07-06 08:04:36.979497+00:00) quotatotalworker stdout | 2023-07-06 08:04:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:04:36,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:04:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:04:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:04:49,861 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:04:50,068 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:04:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:04:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:05:06.979497+00:00 (in 14.999544 seconds) quotatotalworker stdout | 2023-07-06 08:04:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:05:06 UTC)" (scheduled at 2023-07-06 08:04:51.979497+00:00) quotatotalworker stdout | 2023-07-06 08:04:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:04:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:04:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:05:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:05:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:05:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:05:21.979497+00:00 (in 14.999553 seconds) quotatotalworker stdout | 2023-07-06 08:05:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:05:21 UTC)" (scheduled at 2023-07-06 08:05:06.979497+00:00) quotatotalworker stdout | 2023-07-06 08:05:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:05:06,990 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:05:06,990 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:05:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:05:19,893 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:05:20,084 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:05:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:05:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:05:36.979497+00:00 (in 14.999574 seconds) quotatotalworker stdout | 2023-07-06 08:05:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:05:36 UTC)" (scheduled at 2023-07-06 08:05:21.979497+00:00) quotatotalworker stdout | 2023-07-06 08:05:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:05:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:05:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:05:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 08:05:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 08:05:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:06:34.887001+00:00 (in 59.999526 seconds) quotaregistrysizeworker stdout | 2023-07-06 08:05:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:06:34 UTC)" (scheduled at 2023-07-06 08:05:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 08:05:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 08:05:34,895 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 08:05:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:06:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:05:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:05:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:05:51.979497+00:00 (in 14.999572 seconds) quotatotalworker stdout | 2023-07-06 08:05:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:05:51 UTC)" (scheduled at 2023-07-06 08:05:36.979497+00:00) quotatotalworker stdout | 2023-07-06 08:05:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:05:36,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:05:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:05:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:05:49,924 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:05:50,113 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:05:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:05:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:06:06.979497+00:00 (in 14.999582 seconds) quotatotalworker stdout | 2023-07-06 08:05:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:06:06 UTC)" (scheduled at 2023-07-06 08:05:51.979497+00:00) quotatotalworker stdout | 2023-07-06 08:05:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:05:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:05:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:06:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:06:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:06:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:06:21.979497+00:00 (in 14.999560 seconds) quotatotalworker stdout | 2023-07-06 08:06:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:06:21 UTC)" (scheduled at 2023-07-06 08:06:06.979497+00:00) quotatotalworker stdout | 2023-07-06 08:06:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:06:06,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:06:06,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:06:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:06:19,944 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:06:20,145 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:06:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:06:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:06:36.979497+00:00 (in 14.999587 seconds) quotatotalworker stdout | 2023-07-06 08:06:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:06:36 UTC)" (scheduled at 2023-07-06 08:06:21.979497+00:00) quotatotalworker stdout | 2023-07-06 08:06:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:06:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:06:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:06:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 08:06:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 08:06:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:07:34.887001+00:00 (in 59.999498 seconds) quotaregistrysizeworker stdout | 2023-07-06 08:06:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:07:34 UTC)" (scheduled at 2023-07-06 08:06:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 08:06:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 08:06:34,895 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 08:06:34,895 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:07:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:06:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:06:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:06:51.979497+00:00 (in 14.999526 seconds) quotatotalworker stdout | 2023-07-06 08:06:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:06:51 UTC)" (scheduled at 2023-07-06 08:06:36.979497+00:00) quotatotalworker stdout | 2023-07-06 08:06:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:06:36,990 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:06:36,990 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:06:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:06:49,968 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:06:50,162 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:06:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:06:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:07:06.979497+00:00 (in 14.999583 seconds) quotatotalworker stdout | 2023-07-06 08:06:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:07:06 UTC)" (scheduled at 2023-07-06 08:06:51.979497+00:00) quotatotalworker stdout | 2023-07-06 08:06:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:06:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:06:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:07:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:07:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:07:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:07:21.979497+00:00 (in 14.999572 seconds) quotatotalworker stdout | 2023-07-06 08:07:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:07:21 UTC)" (scheduled at 2023-07-06 08:07:06.979497+00:00) quotatotalworker stdout | 2023-07-06 08:07:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:07:06,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:07:06,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:07:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:07:19,988 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:07:20,186 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:07:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:07:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:07:36.979497+00:00 (in 14.999575 seconds) quotatotalworker stdout | 2023-07-06 08:07:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:07:36 UTC)" (scheduled at 2023-07-06 08:07:21.979497+00:00) quotatotalworker stdout | 2023-07-06 08:07:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:07:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:07:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:07:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 08:07:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 08:07:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:08:34.887001+00:00 (in 59.999536 seconds) quotaregistrysizeworker stdout | 2023-07-06 08:07:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:08:34 UTC)" (scheduled at 2023-07-06 08:07:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 08:07:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 08:07:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 08:07:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:08:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:07:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:07:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:07:51.979497+00:00 (in 14.999597 seconds) quotatotalworker stdout | 2023-07-06 08:07:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:07:51 UTC)" (scheduled at 2023-07-06 08:07:36.979497+00:00) quotatotalworker stdout | 2023-07-06 08:07:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:07:36,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:07:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:07:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:07:50,007 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:07:50,218 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:07:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:07:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:08:06.979497+00:00 (in 14.999596 seconds) quotatotalworker stdout | 2023-07-06 08:07:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:08:06 UTC)" (scheduled at 2023-07-06 08:07:51.979497+00:00) quotatotalworker stdout | 2023-07-06 08:07:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:07:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:07:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:08:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:08:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:08:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:08:21.979497+00:00 (in 14.999524 seconds) quotatotalworker stdout | 2023-07-06 08:08:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:08:21 UTC)" (scheduled at 2023-07-06 08:08:06.979497+00:00) quotatotalworker stdout | 2023-07-06 08:08:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:08:06,990 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:08:06,990 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:08:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:08:20,012 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:08:20,251 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:08:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:08:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:08:36.979497+00:00 (in 14.999548 seconds) quotatotalworker stdout | 2023-07-06 08:08:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:08:36 UTC)" (scheduled at 2023-07-06 08:08:21.979497+00:00) quotatotalworker stdout | 2023-07-06 08:08:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:08:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:08:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:08:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 08:08:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 08:08:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:09:34.887001+00:00 (in 59.999520 seconds) quotaregistrysizeworker stdout | 2023-07-06 08:08:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:09:34 UTC)" (scheduled at 2023-07-06 08:08:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 08:08:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 08:08:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 08:08:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:09:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:08:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:08:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:08:51.979497+00:00 (in 14.999546 seconds) quotatotalworker stdout | 2023-07-06 08:08:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:08:51 UTC)" (scheduled at 2023-07-06 08:08:36.979497+00:00) quotatotalworker stdout | 2023-07-06 08:08:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:08:36,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:08:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:08:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:08:50,032 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:08:50,283 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:08:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:08:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:09:06.979497+00:00 (in 14.999580 seconds) quotatotalworker stdout | 2023-07-06 08:08:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:09:06 UTC)" (scheduled at 2023-07-06 08:08:51.979497+00:00) quotatotalworker stdout | 2023-07-06 08:08:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:08:51,987 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:08:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:09:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:09:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:09:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:09:21.979497+00:00 (in 14.999539 seconds) quotatotalworker stdout | 2023-07-06 08:09:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:09:21 UTC)" (scheduled at 2023-07-06 08:09:06.979497+00:00) quotatotalworker stdout | 2023-07-06 08:09:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:09:06,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:09:06,990 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:09:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:09:20,059 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:09:20,310 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:09:21,980 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:09:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:09:36.979497+00:00 (in 14.999378 seconds) quotatotalworker stdout | 2023-07-06 08:09:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:09:36 UTC)" (scheduled at 2023-07-06 08:09:21.979497+00:00) quotatotalworker stdout | 2023-07-06 08:09:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:09:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:09:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:09:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 08:09:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 08:09:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:10:34.887001+00:00 (in 59.999538 seconds) quotaregistrysizeworker stdout | 2023-07-06 08:09:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:10:34 UTC)" (scheduled at 2023-07-06 08:09:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 08:09:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 08:09:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 08:09:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:10:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:09:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:09:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:09:51.979497+00:00 (in 14.999497 seconds) quotatotalworker stdout | 2023-07-06 08:09:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:09:51 UTC)" (scheduled at 2023-07-06 08:09:36.979497+00:00) quotatotalworker stdout | 2023-07-06 08:09:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:09:36,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:09:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:09:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:09:50,091 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:09:50,338 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:09:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:09:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:10:06.979497+00:00 (in 14.999471 seconds) quotatotalworker stdout | 2023-07-06 08:09:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:10:06 UTC)" (scheduled at 2023-07-06 08:09:51.979497+00:00) quotatotalworker stdout | 2023-07-06 08:09:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:09:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:09:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:10:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:10:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:10:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:10:21.979497+00:00 (in 14.999584 seconds) quotatotalworker stdout | 2023-07-06 08:10:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:10:21 UTC)" (scheduled at 2023-07-06 08:10:06.979497+00:00) quotatotalworker stdout | 2023-07-06 08:10:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:10:06,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:10:06,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:10:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:10:20,124 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:10:20,371 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:10:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:10:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:10:36.979497+00:00 (in 14.999573 seconds) quotatotalworker stdout | 2023-07-06 08:10:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:10:36 UTC)" (scheduled at 2023-07-06 08:10:21.979497+00:00) quotatotalworker stdout | 2023-07-06 08:10:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:10:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:10:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:10:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 08:10:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 08:10:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:11:34.887001+00:00 (in 59.999559 seconds) quotaregistrysizeworker stdout | 2023-07-06 08:10:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:11:34 UTC)" (scheduled at 2023-07-06 08:10:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 08:10:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 08:10:34,895 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 08:10:34,895 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:11:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:10:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:10:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:10:51.979497+00:00 (in 14.999520 seconds) quotatotalworker stdout | 2023-07-06 08:10:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:10:51 UTC)" (scheduled at 2023-07-06 08:10:36.979497+00:00) quotatotalworker stdout | 2023-07-06 08:10:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:10:36,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:10:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:10:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:10:50,148 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:10:50,403 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:10:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:10:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:11:06.979497+00:00 (in 14.999565 seconds) quotatotalworker stdout | 2023-07-06 08:10:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:11:06 UTC)" (scheduled at 2023-07-06 08:10:51.979497+00:00) quotatotalworker stdout | 2023-07-06 08:10:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:10:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:10:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:11:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:11:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:11:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:11:21.979497+00:00 (in 14.999557 seconds) quotatotalworker stdout | 2023-07-06 08:11:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:11:21 UTC)" (scheduled at 2023-07-06 08:11:06.979497+00:00) quotatotalworker stdout | 2023-07-06 08:11:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:11:06,990 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:11:06,990 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:11:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:11:20,165 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:11:20,435 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:11:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:11:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:11:36.979497+00:00 (in 14.999573 seconds) quotatotalworker stdout | 2023-07-06 08:11:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:11:36 UTC)" (scheduled at 2023-07-06 08:11:21.979497+00:00) quotatotalworker stdout | 2023-07-06 08:11:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:11:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:11:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:11:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 08:11:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 08:11:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:12:34.887001+00:00 (in 59.999569 seconds) quotaregistrysizeworker stdout | 2023-07-06 08:11:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:12:34 UTC)" (scheduled at 2023-07-06 08:11:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 08:11:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 08:11:34,895 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 08:11:34,895 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:12:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:11:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:11:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:11:51.979497+00:00 (in 14.999553 seconds) quotatotalworker stdout | 2023-07-06 08:11:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:11:51 UTC)" (scheduled at 2023-07-06 08:11:36.979497+00:00) quotatotalworker stdout | 2023-07-06 08:11:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:11:36,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:11:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:11:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:11:50,197 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:11:50,456 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:11:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:11:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:12:06.979497+00:00 (in 14.999556 seconds) quotatotalworker stdout | 2023-07-06 08:11:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:12:06 UTC)" (scheduled at 2023-07-06 08:11:51.979497+00:00) quotatotalworker stdout | 2023-07-06 08:11:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:11:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:11:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:12:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:12:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:12:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:12:21.979497+00:00 (in 14.999581 seconds) quotatotalworker stdout | 2023-07-06 08:12:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:12:21 UTC)" (scheduled at 2023-07-06 08:12:06.979497+00:00) quotatotalworker stdout | 2023-07-06 08:12:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:12:06,990 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:12:06,990 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:12:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:12:20,229 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:12:20,488 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:12:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:12:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:12:36.979497+00:00 (in 14.999549 seconds) quotatotalworker stdout | 2023-07-06 08:12:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:12:36 UTC)" (scheduled at 2023-07-06 08:12:21.979497+00:00) quotatotalworker stdout | 2023-07-06 08:12:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:12:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:12:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:12:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 08:12:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 08:12:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:13:34.887001+00:00 (in 59.999550 seconds) quotaregistrysizeworker stdout | 2023-07-06 08:12:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:13:34 UTC)" (scheduled at 2023-07-06 08:12:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 08:12:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 08:12:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 08:12:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:13:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:12:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:12:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:12:51.979497+00:00 (in 14.999610 seconds) quotatotalworker stdout | 2023-07-06 08:12:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:12:51 UTC)" (scheduled at 2023-07-06 08:12:36.979497+00:00) quotatotalworker stdout | 2023-07-06 08:12:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:12:36,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:12:36,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:12:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:12:50,261 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:12:50,494 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:12:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:12:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:13:06.979497+00:00 (in 14.999562 seconds) quotatotalworker stdout | 2023-07-06 08:12:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:13:06 UTC)" (scheduled at 2023-07-06 08:12:51.979497+00:00) quotatotalworker stdout | 2023-07-06 08:12:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:12:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:12:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:13:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:13:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:13:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:13:21.979497+00:00 (in 14.999542 seconds) quotatotalworker stdout | 2023-07-06 08:13:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:13:21 UTC)" (scheduled at 2023-07-06 08:13:06.979497+00:00) quotatotalworker stdout | 2023-07-06 08:13:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:13:06,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:13:06,990 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:13:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:13:20,281 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:13:20,505 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:13:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:13:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:13:36.979497+00:00 (in 14.999549 seconds) quotatotalworker stdout | 2023-07-06 08:13:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:13:36 UTC)" (scheduled at 2023-07-06 08:13:21.979497+00:00) quotatotalworker stdout | 2023-07-06 08:13:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:13:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:13:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:13:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 08:13:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 08:13:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:14:34.887001+00:00 (in 59.999469 seconds) quotaregistrysizeworker stdout | 2023-07-06 08:13:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:14:34 UTC)" (scheduled at 2023-07-06 08:13:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 08:13:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 08:13:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 08:13:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:14:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:13:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:13:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:13:51.979497+00:00 (in 14.999510 seconds) quotatotalworker stdout | 2023-07-06 08:13:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:13:51 UTC)" (scheduled at 2023-07-06 08:13:36.979497+00:00) quotatotalworker stdout | 2023-07-06 08:13:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:13:36,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:13:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:13:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:13:50,313 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:13:50,537 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:13:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:13:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:14:06.979497+00:00 (in 14.999482 seconds) quotatotalworker stdout | 2023-07-06 08:13:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:14:06 UTC)" (scheduled at 2023-07-06 08:13:51.979497+00:00) quotatotalworker stdout | 2023-07-06 08:13:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:13:51,987 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:13:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:14:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:14:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:14:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:14:21.979497+00:00 (in 14.999432 seconds) quotatotalworker stdout | 2023-07-06 08:14:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:14:21 UTC)" (scheduled at 2023-07-06 08:14:06.979497+00:00) quotatotalworker stdout | 2023-07-06 08:14:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:14:06,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:14:06,990 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:14:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:14:20,320 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:14:20,553 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:14:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:14:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:14:36.979497+00:00 (in 14.999477 seconds) quotatotalworker stdout | 2023-07-06 08:14:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:14:36 UTC)" (scheduled at 2023-07-06 08:14:21.979497+00:00) quotatotalworker stdout | 2023-07-06 08:14:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:14:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:14:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:14:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 08:14:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 08:14:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:15:34.887001+00:00 (in 59.999560 seconds) quotaregistrysizeworker stdout | 2023-07-06 08:14:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:15:34 UTC)" (scheduled at 2023-07-06 08:14:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 08:14:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 08:14:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 08:14:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:15:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:14:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:14:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:14:51.979497+00:00 (in 14.999445 seconds) quotatotalworker stdout | 2023-07-06 08:14:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:14:51 UTC)" (scheduled at 2023-07-06 08:14:36.979497+00:00) quotatotalworker stdout | 2023-07-06 08:14:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:14:36,994 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:14:36,994 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:14:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:14:50,352 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:14:50,574 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:14:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:14:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:15:06.979497+00:00 (in 14.999544 seconds) quotatotalworker stdout | 2023-07-06 08:14:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:15:06 UTC)" (scheduled at 2023-07-06 08:14:51.979497+00:00) quotatotalworker stdout | 2023-07-06 08:14:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:14:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:14:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:15:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:15:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:15:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:15:21.979497+00:00 (in 14.999570 seconds) quotatotalworker stdout | 2023-07-06 08:15:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:15:21 UTC)" (scheduled at 2023-07-06 08:15:06.979497+00:00) quotatotalworker stdout | 2023-07-06 08:15:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:15:06,994 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:15:06,994 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:15:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:15:20,371 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:15:20,579 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:15:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:15:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:15:36.979497+00:00 (in 14.999572 seconds) quotatotalworker stdout | 2023-07-06 08:15:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:15:36 UTC)" (scheduled at 2023-07-06 08:15:21.979497+00:00) quotatotalworker stdout | 2023-07-06 08:15:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:15:21,991 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:15:21,991 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:15:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 08:15:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 08:15:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:16:34.887001+00:00 (in 59.999512 seconds) quotaregistrysizeworker stdout | 2023-07-06 08:15:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:16:34 UTC)" (scheduled at 2023-07-06 08:15:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 08:15:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 08:15:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 08:15:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:16:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:15:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:15:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:15:51.979497+00:00 (in 14.999587 seconds) quotatotalworker stdout | 2023-07-06 08:15:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:15:51 UTC)" (scheduled at 2023-07-06 08:15:36.979497+00:00) quotatotalworker stdout | 2023-07-06 08:15:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:15:36,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:15:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:15:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:15:50,377 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:15:50,586 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:15:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:15:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:16:06.979497+00:00 (in 14.999585 seconds) quotatotalworker stdout | 2023-07-06 08:15:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:16:06 UTC)" (scheduled at 2023-07-06 08:15:51.979497+00:00) quotatotalworker stdout | 2023-07-06 08:15:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:15:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:15:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:16:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:16:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:16:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:16:21.979497+00:00 (in 14.999582 seconds) quotatotalworker stdout | 2023-07-06 08:16:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:16:21 UTC)" (scheduled at 2023-07-06 08:16:06.979497+00:00) quotatotalworker stdout | 2023-07-06 08:16:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:16:06,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:16:06,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:16:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:16:20,391 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:16:20,618 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:16:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:16:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:16:36.979497+00:00 (in 14.999607 seconds) quotatotalworker stdout | 2023-07-06 08:16:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:16:36 UTC)" (scheduled at 2023-07-06 08:16:21.979497+00:00) quotatotalworker stdout | 2023-07-06 08:16:21,980 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:16:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:16:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:16:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 08:16:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 08:16:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:17:34.887001+00:00 (in 59.999468 seconds) quotaregistrysizeworker stdout | 2023-07-06 08:16:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:17:34 UTC)" (scheduled at 2023-07-06 08:16:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 08:16:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 08:16:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 08:16:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:17:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:16:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:16:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:16:51.979497+00:00 (in 14.999578 seconds) quotatotalworker stdout | 2023-07-06 08:16:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:16:51 UTC)" (scheduled at 2023-07-06 08:16:36.979497+00:00) quotatotalworker stdout | 2023-07-06 08:16:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:16:37,000 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:16:37,000 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:16:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:16:50,418 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:16:50,622 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:16:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:16:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:17:06.979497+00:00 (in 14.999592 seconds) quotatotalworker stdout | 2023-07-06 08:16:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:17:06 UTC)" (scheduled at 2023-07-06 08:16:51.979497+00:00) quotatotalworker stdout | 2023-07-06 08:16:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:16:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:16:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:17:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:17:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:17:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:17:21.979497+00:00 (in 14.999553 seconds) quotatotalworker stdout | 2023-07-06 08:17:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:17:21 UTC)" (scheduled at 2023-07-06 08:17:06.979497+00:00) quotatotalworker stdout | 2023-07-06 08:17:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:17:06,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:17:06,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:17:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:17:20,431 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:17:20,625 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:17:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:17:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:17:36.979497+00:00 (in 14.999554 seconds) quotatotalworker stdout | 2023-07-06 08:17:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:17:36 UTC)" (scheduled at 2023-07-06 08:17:21.979497+00:00) quotatotalworker stdout | 2023-07-06 08:17:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:17:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:17:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:17:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 08:17:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 08:17:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:18:34.887001+00:00 (in 59.999445 seconds) quotaregistrysizeworker stdout | 2023-07-06 08:17:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:18:34 UTC)" (scheduled at 2023-07-06 08:17:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 08:17:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 08:17:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 08:17:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:18:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:17:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:17:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:17:51.979497+00:00 (in 14.999518 seconds) quotatotalworker stdout | 2023-07-06 08:17:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:17:51 UTC)" (scheduled at 2023-07-06 08:17:36.979497+00:00) quotatotalworker stdout | 2023-07-06 08:17:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:17:36,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:17:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:17:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:17:50,456 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:17:50,657 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:17:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:17:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:18:06.979497+00:00 (in 14.999575 seconds) quotatotalworker stdout | 2023-07-06 08:17:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:18:06 UTC)" (scheduled at 2023-07-06 08:17:51.979497+00:00) quotatotalworker stdout | 2023-07-06 08:17:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:17:51,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:17:51,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:18:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:18:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:18:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:18:21.979497+00:00 (in 14.999566 seconds) quotatotalworker stdout | 2023-07-06 08:18:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:18:21 UTC)" (scheduled at 2023-07-06 08:18:06.979497+00:00) quotatotalworker stdout | 2023-07-06 08:18:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:18:06,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:18:06,990 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:18:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:18:20,488 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:18:20,689 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:18:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:18:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:18:36.979497+00:00 (in 14.999536 seconds) quotatotalworker stdout | 2023-07-06 08:18:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:18:36 UTC)" (scheduled at 2023-07-06 08:18:21.979497+00:00) quotatotalworker stdout | 2023-07-06 08:18:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:18:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:18:21,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:18:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 08:18:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 08:18:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:19:34.887001+00:00 (in 59.999479 seconds) quotaregistrysizeworker stdout | 2023-07-06 08:18:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:19:34 UTC)" (scheduled at 2023-07-06 08:18:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 08:18:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 08:18:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 08:18:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:19:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:18:36,980 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:18:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:18:51.979497+00:00 (in 14.999046 seconds) quotatotalworker stdout | 2023-07-06 08:18:37,006 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:18:51 UTC)" (scheduled at 2023-07-06 08:18:36.979497+00:00) quotatotalworker stdout | 2023-07-06 08:18:37,007 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:18:37,015 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:18:37,015 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:18:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:18:50,508 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:18:50,694 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:18:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:18:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:19:06.979497+00:00 (in 14.999458 seconds) quotatotalworker stdout | 2023-07-06 08:18:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:19:06 UTC)" (scheduled at 2023-07-06 08:18:51.979497+00:00) quotatotalworker stdout | 2023-07-06 08:18:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:18:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:18:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:19:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:19:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:19:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:19:21.979497+00:00 (in 14.999499 seconds) quotatotalworker stdout | 2023-07-06 08:19:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:19:21 UTC)" (scheduled at 2023-07-06 08:19:06.979497+00:00) quotatotalworker stdout | 2023-07-06 08:19:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:19:06,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:19:06,990 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:19:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:19:20,532 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:19:20,696 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:19:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:19:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:19:36.979497+00:00 (in 14.999437 seconds) quotatotalworker stdout | 2023-07-06 08:19:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:19:36 UTC)" (scheduled at 2023-07-06 08:19:21.979497+00:00) quotatotalworker stdout | 2023-07-06 08:19:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:19:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:19:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:19:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 08:19:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 08:19:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:20:34.887001+00:00 (in 59.999450 seconds) quotaregistrysizeworker stdout | 2023-07-06 08:19:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:20:34 UTC)" (scheduled at 2023-07-06 08:19:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 08:19:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 08:19:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 08:19:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:20:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:19:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:19:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:19:51.979497+00:00 (in 14.999552 seconds) quotatotalworker stdout | 2023-07-06 08:19:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:19:51 UTC)" (scheduled at 2023-07-06 08:19:36.979497+00:00) quotatotalworker stdout | 2023-07-06 08:19:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:19:36,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:19:36,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:19:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:19:50,564 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:19:50,714 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:19:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:19:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:20:06.979497+00:00 (in 14.999586 seconds) quotatotalworker stdout | 2023-07-06 08:19:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:20:06 UTC)" (scheduled at 2023-07-06 08:19:51.979497+00:00) quotatotalworker stdout | 2023-07-06 08:19:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:19:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:19:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:20:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:20:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:20:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:20:21.979497+00:00 (in 14.999518 seconds) quotatotalworker stdout | 2023-07-06 08:20:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:20:21 UTC)" (scheduled at 2023-07-06 08:20:06.979497+00:00) quotatotalworker stdout | 2023-07-06 08:20:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:20:06,990 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:20:06,990 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:20:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:20:20,595 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:20:20,739 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:20:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:20:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:20:36.979497+00:00 (in 14.999474 seconds) quotatotalworker stdout | 2023-07-06 08:20:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:20:36 UTC)" (scheduled at 2023-07-06 08:20:21.979497+00:00) quotatotalworker stdout | 2023-07-06 08:20:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:20:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:20:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:20:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 08:20:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 08:20:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:21:34.887001+00:00 (in 59.999448 seconds) quotaregistrysizeworker stdout | 2023-07-06 08:20:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:21:34 UTC)" (scheduled at 2023-07-06 08:20:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 08:20:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 08:20:34,895 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 08:20:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:21:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:20:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:20:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:20:51.979497+00:00 (in 14.999465 seconds) quotatotalworker stdout | 2023-07-06 08:20:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:20:51 UTC)" (scheduled at 2023-07-06 08:20:36.979497+00:00) quotatotalworker stdout | 2023-07-06 08:20:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:20:36,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:20:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:20:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:20:50,597 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:20:50,748 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:20:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:20:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:21:06.979497+00:00 (in 14.999497 seconds) quotatotalworker stdout | 2023-07-06 08:20:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:21:06 UTC)" (scheduled at 2023-07-06 08:20:51.979497+00:00) quotatotalworker stdout | 2023-07-06 08:20:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:20:51,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:20:51,990 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:21:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:21:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:21:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:21:21.979497+00:00 (in 14.999517 seconds) quotatotalworker stdout | 2023-07-06 08:21:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:21:21 UTC)" (scheduled at 2023-07-06 08:21:06.979497+00:00) quotatotalworker stdout | 2023-07-06 08:21:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:21:06,990 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:21:06,990 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:21:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:21:20,629 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:21:20,780 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:21:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:21:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:21:36.979497+00:00 (in 14.999480 seconds) quotatotalworker stdout | 2023-07-06 08:21:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:21:36 UTC)" (scheduled at 2023-07-06 08:21:21.979497+00:00) quotatotalworker stdout | 2023-07-06 08:21:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:21:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:21:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:21:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 08:21:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 08:21:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:22:34.887001+00:00 (in 59.999466 seconds) quotaregistrysizeworker stdout | 2023-07-06 08:21:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:22:34 UTC)" (scheduled at 2023-07-06 08:21:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 08:21:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 08:21:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 08:21:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:22:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:21:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:21:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:21:51.979497+00:00 (in 14.999464 seconds) quotatotalworker stdout | 2023-07-06 08:21:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:21:51 UTC)" (scheduled at 2023-07-06 08:21:36.979497+00:00) quotatotalworker stdout | 2023-07-06 08:21:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:21:36,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:21:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:21:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:21:50,660 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:21:50,793 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:21:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:21:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:22:06.979497+00:00 (in 14.999542 seconds) quotatotalworker stdout | 2023-07-06 08:21:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:22:06 UTC)" (scheduled at 2023-07-06 08:21:51.979497+00:00) quotatotalworker stdout | 2023-07-06 08:21:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:21:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:21:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:22:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:22:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:22:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:22:21.979497+00:00 (in 14.999515 seconds) quotatotalworker stdout | 2023-07-06 08:22:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:22:21 UTC)" (scheduled at 2023-07-06 08:22:06.979497+00:00) quotatotalworker stdout | 2023-07-06 08:22:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:22:06,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:22:06,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:22:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:22:20,689 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:22:20,825 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:22:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:22:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:22:36.979497+00:00 (in 14.999580 seconds) quotatotalworker stdout | 2023-07-06 08:22:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:22:36 UTC)" (scheduled at 2023-07-06 08:22:21.979497+00:00) quotatotalworker stdout | 2023-07-06 08:22:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:22:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:22:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:22:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 08:22:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 08:22:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:23:34.887001+00:00 (in 59.999480 seconds) quotaregistrysizeworker stdout | 2023-07-06 08:22:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:23:34 UTC)" (scheduled at 2023-07-06 08:22:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 08:22:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 08:22:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 08:22:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:23:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:22:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:22:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:22:51.979497+00:00 (in 14.999509 seconds) quotatotalworker stdout | 2023-07-06 08:22:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:22:51 UTC)" (scheduled at 2023-07-06 08:22:36.979497+00:00) quotatotalworker stdout | 2023-07-06 08:22:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:22:36,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:22:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:22:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:22:50,708 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:22:50,857 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:22:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:22:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:23:06.979497+00:00 (in 14.999569 seconds) quotatotalworker stdout | 2023-07-06 08:22:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:23:06 UTC)" (scheduled at 2023-07-06 08:22:51.979497+00:00) quotatotalworker stdout | 2023-07-06 08:22:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:22:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:22:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:23:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:23:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:23:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:23:21.979497+00:00 (in 14.999486 seconds) quotatotalworker stdout | 2023-07-06 08:23:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:23:21 UTC)" (scheduled at 2023-07-06 08:23:06.979497+00:00) quotatotalworker stdout | 2023-07-06 08:23:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:23:06,990 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:23:06,990 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:23:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:23:20,740 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:23:20,889 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:23:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:23:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:23:36.979497+00:00 (in 14.999553 seconds) quotatotalworker stdout | 2023-07-06 08:23:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:23:36 UTC)" (scheduled at 2023-07-06 08:23:21.979497+00:00) quotatotalworker stdout | 2023-07-06 08:23:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:23:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:23:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:23:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 08:23:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 08:23:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:24:34.887001+00:00 (in 59.999523 seconds) quotaregistrysizeworker stdout | 2023-07-06 08:23:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:24:34 UTC)" (scheduled at 2023-07-06 08:23:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 08:23:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 08:23:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 08:23:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:24:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:23:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:23:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:23:51.979497+00:00 (in 14.999494 seconds) quotatotalworker stdout | 2023-07-06 08:23:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:23:51 UTC)" (scheduled at 2023-07-06 08:23:36.979497+00:00) quotatotalworker stdout | 2023-07-06 08:23:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:23:37,002 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:23:37,002 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:23:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:23:50,756 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:23:50,914 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:23:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:23:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:24:06.979497+00:00 (in 14.999504 seconds) quotatotalworker stdout | 2023-07-06 08:23:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:24:06 UTC)" (scheduled at 2023-07-06 08:23:51.979497+00:00) quotatotalworker stdout | 2023-07-06 08:23:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:23:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:23:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:24:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:24:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:24:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:24:21.979497+00:00 (in 14.999537 seconds) quotatotalworker stdout | 2023-07-06 08:24:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:24:21 UTC)" (scheduled at 2023-07-06 08:24:06.979497+00:00) quotatotalworker stdout | 2023-07-06 08:24:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:24:06,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:24:06,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:24:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:24:20,759 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:24:20,944 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:24:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:24:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:24:36.979497+00:00 (in 14.999551 seconds) quotatotalworker stdout | 2023-07-06 08:24:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:24:36 UTC)" (scheduled at 2023-07-06 08:24:21.979497+00:00) quotatotalworker stdout | 2023-07-06 08:24:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:24:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:24:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:24:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 08:24:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 08:24:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:25:34.887001+00:00 (in 59.999527 seconds) quotaregistrysizeworker stdout | 2023-07-06 08:24:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:25:34 UTC)" (scheduled at 2023-07-06 08:24:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 08:24:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 08:24:34,895 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 08:24:34,895 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:25:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:24:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:24:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:24:51.979497+00:00 (in 14.999531 seconds) quotatotalworker stdout | 2023-07-06 08:24:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:24:51 UTC)" (scheduled at 2023-07-06 08:24:36.979497+00:00) quotatotalworker stdout | 2023-07-06 08:24:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:24:36,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:24:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:24:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:24:50,791 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:24:50,958 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:24:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:24:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:25:06.979497+00:00 (in 14.999510 seconds) quotatotalworker stdout | 2023-07-06 08:24:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:25:06 UTC)" (scheduled at 2023-07-06 08:24:51.979497+00:00) quotatotalworker stdout | 2023-07-06 08:24:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:24:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:24:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:25:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:25:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:25:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:25:21.979497+00:00 (in 14.999494 seconds) quotatotalworker stdout | 2023-07-06 08:25:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:25:21 UTC)" (scheduled at 2023-07-06 08:25:06.979497+00:00) quotatotalworker stdout | 2023-07-06 08:25:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:25:06,990 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:25:06,990 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:25:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:25:20,824 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:25:20,987 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:25:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:25:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:25:36.979497+00:00 (in 14.999460 seconds) quotatotalworker stdout | 2023-07-06 08:25:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:25:36 UTC)" (scheduled at 2023-07-06 08:25:21.979497+00:00) quotatotalworker stdout | 2023-07-06 08:25:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:25:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:25:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:25:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 08:25:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 08:25:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:26:34.887001+00:00 (in 59.999564 seconds) quotaregistrysizeworker stdout | 2023-07-06 08:25:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:26:34 UTC)" (scheduled at 2023-07-06 08:25:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 08:25:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 08:25:34,895 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 08:25:34,895 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:26:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:25:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:25:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:25:51.979497+00:00 (in 14.999513 seconds) quotatotalworker stdout | 2023-07-06 08:25:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:25:51 UTC)" (scheduled at 2023-07-06 08:25:36.979497+00:00) quotatotalworker stdout | 2023-07-06 08:25:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:25:36,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:25:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:25:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:25:50,856 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:25:51,018 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:25:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:25:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:26:06.979497+00:00 (in 14.999524 seconds) quotatotalworker stdout | 2023-07-06 08:25:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:26:06 UTC)" (scheduled at 2023-07-06 08:25:51.979497+00:00) quotatotalworker stdout | 2023-07-06 08:25:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:25:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:25:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:26:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:26:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:26:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:26:21.979497+00:00 (in 14.999561 seconds) quotatotalworker stdout | 2023-07-06 08:26:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:26:21 UTC)" (scheduled at 2023-07-06 08:26:06.979497+00:00) quotatotalworker stdout | 2023-07-06 08:26:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:26:06,990 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:26:06,990 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:26:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:26:20,867 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:26:21,023 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:26:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:26:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:26:36.979497+00:00 (in 14.999593 seconds) quotatotalworker stdout | 2023-07-06 08:26:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:26:36 UTC)" (scheduled at 2023-07-06 08:26:21.979497+00:00) quotatotalworker stdout | 2023-07-06 08:26:21,980 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:26:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:26:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:26:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 08:26:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 08:26:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:27:34.887001+00:00 (in 59.999196 seconds) quotaregistrysizeworker stdout | 2023-07-06 08:26:34,888 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:27:34 UTC)" (scheduled at 2023-07-06 08:26:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 08:26:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 08:26:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 08:26:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:27:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:26:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:26:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:26:51.979497+00:00 (in 14.999537 seconds) quotatotalworker stdout | 2023-07-06 08:26:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:26:51 UTC)" (scheduled at 2023-07-06 08:26:36.979497+00:00) quotatotalworker stdout | 2023-07-06 08:26:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:26:36,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:26:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:26:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:26:50,876 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:26:51,055 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:26:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:26:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:27:06.979497+00:00 (in 14.999573 seconds) quotatotalworker stdout | 2023-07-06 08:26:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:27:06 UTC)" (scheduled at 2023-07-06 08:26:51.979497+00:00) quotatotalworker stdout | 2023-07-06 08:26:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:26:51,990 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:26:51,990 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:27:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:27:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:27:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:27:21.979497+00:00 (in 14.999559 seconds) quotatotalworker stdout | 2023-07-06 08:27:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:27:21 UTC)" (scheduled at 2023-07-06 08:27:06.979497+00:00) quotatotalworker stdout | 2023-07-06 08:27:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:27:06,990 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:27:06,990 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:27:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:27:20,906 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:27:21,088 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:27:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:27:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:27:36.979497+00:00 (in 14.999574 seconds) quotatotalworker stdout | 2023-07-06 08:27:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:27:36 UTC)" (scheduled at 2023-07-06 08:27:21.979497+00:00) quotatotalworker stdout | 2023-07-06 08:27:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:27:21,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:27:21,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:27:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 08:27:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 08:27:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:28:34.887001+00:00 (in 59.999554 seconds) quotaregistrysizeworker stdout | 2023-07-06 08:27:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:28:34 UTC)" (scheduled at 2023-07-06 08:27:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 08:27:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 08:27:34,895 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 08:27:34,895 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:28:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:27:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:27:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:27:51.979497+00:00 (in 14.999538 seconds) quotatotalworker stdout | 2023-07-06 08:27:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:27:51 UTC)" (scheduled at 2023-07-06 08:27:36.979497+00:00) quotatotalworker stdout | 2023-07-06 08:27:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:27:36,990 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:27:36,991 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:27:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:27:50,915 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:27:51,120 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:27:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:27:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:28:06.979497+00:00 (in 14.999553 seconds) quotatotalworker stdout | 2023-07-06 08:27:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:28:06 UTC)" (scheduled at 2023-07-06 08:27:51.979497+00:00) quotatotalworker stdout | 2023-07-06 08:27:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:27:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:27:51,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:28:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:28:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:28:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:28:21.979497+00:00 (in 14.999573 seconds) quotatotalworker stdout | 2023-07-06 08:28:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:28:21 UTC)" (scheduled at 2023-07-06 08:28:06.979497+00:00) quotatotalworker stdout | 2023-07-06 08:28:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:28:06,990 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:28:06,990 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:28:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:28:20,923 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:28:21,152 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:28:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:28:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:28:36.979497+00:00 (in 14.999526 seconds) quotatotalworker stdout | 2023-07-06 08:28:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:28:36 UTC)" (scheduled at 2023-07-06 08:28:21.979497+00:00) quotatotalworker stdout | 2023-07-06 08:28:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:28:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:28:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:28:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 08:28:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 08:28:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:29:34.887001+00:00 (in 59.999508 seconds) quotaregistrysizeworker stdout | 2023-07-06 08:28:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:29:34 UTC)" (scheduled at 2023-07-06 08:28:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 08:28:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 08:28:34,895 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 08:28:34,895 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:29:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:28:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:28:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:28:51.979497+00:00 (in 14.999542 seconds) quotatotalworker stdout | 2023-07-06 08:28:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:28:51 UTC)" (scheduled at 2023-07-06 08:28:36.979497+00:00) quotatotalworker stdout | 2023-07-06 08:28:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:28:36,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:28:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:28:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:28:50,950 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:28:51,182 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:28:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:28:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:29:06.979497+00:00 (in 14.999568 seconds) quotatotalworker stdout | 2023-07-06 08:28:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:29:06 UTC)" (scheduled at 2023-07-06 08:28:51.979497+00:00) quotatotalworker stdout | 2023-07-06 08:28:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:28:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:28:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:29:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:29:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:29:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:29:21.979497+00:00 (in 14.999546 seconds) quotatotalworker stdout | 2023-07-06 08:29:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:29:21 UTC)" (scheduled at 2023-07-06 08:29:06.979497+00:00) quotatotalworker stdout | 2023-07-06 08:29:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:29:06,990 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:29:06,990 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:29:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:29:20,982 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:29:21,187 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:29:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:29:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:29:36.979497+00:00 (in 14.999575 seconds) quotatotalworker stdout | 2023-07-06 08:29:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:29:36 UTC)" (scheduled at 2023-07-06 08:29:21.979497+00:00) quotatotalworker stdout | 2023-07-06 08:29:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:29:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:29:21,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:29:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 08:29:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 08:29:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:30:34.887001+00:00 (in 59.999473 seconds) quotaregistrysizeworker stdout | 2023-07-06 08:29:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:30:34 UTC)" (scheduled at 2023-07-06 08:29:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 08:29:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 08:29:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 08:29:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:30:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:29:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:29:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:29:51.979497+00:00 (in 14.999515 seconds) quotatotalworker stdout | 2023-07-06 08:29:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:29:51 UTC)" (scheduled at 2023-07-06 08:29:36.979497+00:00) quotatotalworker stdout | 2023-07-06 08:29:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:29:36,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:29:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:29:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:29:51,004 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:29:51,220 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:29:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:29:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:30:06.979497+00:00 (in 14.999513 seconds) quotatotalworker stdout | 2023-07-06 08:29:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:30:06 UTC)" (scheduled at 2023-07-06 08:29:51.979497+00:00) quotatotalworker stdout | 2023-07-06 08:29:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:29:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:29:51,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:30:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:30:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:30:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:30:21.979497+00:00 (in 14.999513 seconds) quotatotalworker stdout | 2023-07-06 08:30:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:30:21 UTC)" (scheduled at 2023-07-06 08:30:06.979497+00:00) quotatotalworker stdout | 2023-07-06 08:30:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:30:06,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:30:06,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:30:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:30:21,036 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:30:21,252 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:30:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:30:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:30:36.979497+00:00 (in 14.999588 seconds) quotatotalworker stdout | 2023-07-06 08:30:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:30:36 UTC)" (scheduled at 2023-07-06 08:30:21.979497+00:00) quotatotalworker stdout | 2023-07-06 08:30:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:30:21,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:30:21,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:30:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 08:30:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 08:30:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:31:34.887001+00:00 (in 59.999505 seconds) quotaregistrysizeworker stdout | 2023-07-06 08:30:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:31:34 UTC)" (scheduled at 2023-07-06 08:30:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 08:30:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 08:30:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 08:30:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:31:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:30:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:30:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:30:51.979497+00:00 (in 14.999546 seconds) quotatotalworker stdout | 2023-07-06 08:30:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:30:51 UTC)" (scheduled at 2023-07-06 08:30:36.979497+00:00) quotatotalworker stdout | 2023-07-06 08:30:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:30:36,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:30:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:30:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:30:51,045 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:30:51,274 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:30:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:30:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:31:06.979497+00:00 (in 14.999590 seconds) quotatotalworker stdout | 2023-07-06 08:30:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:31:06 UTC)" (scheduled at 2023-07-06 08:30:51.979497+00:00) quotatotalworker stdout | 2023-07-06 08:30:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:30:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:30:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:31:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:31:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:31:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:31:21.979497+00:00 (in 14.999549 seconds) quotatotalworker stdout | 2023-07-06 08:31:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:31:21 UTC)" (scheduled at 2023-07-06 08:31:06.979497+00:00) quotatotalworker stdout | 2023-07-06 08:31:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:31:06,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:31:06,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:31:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:31:21,047 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:31:21,307 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:31:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:31:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:31:36.979497+00:00 (in 14.999534 seconds) quotatotalworker stdout | 2023-07-06 08:31:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:31:36 UTC)" (scheduled at 2023-07-06 08:31:21.979497+00:00) quotatotalworker stdout | 2023-07-06 08:31:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:31:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:31:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:31:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 08:31:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 08:31:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:32:34.887001+00:00 (in 59.999521 seconds) quotaregistrysizeworker stdout | 2023-07-06 08:31:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:32:34 UTC)" (scheduled at 2023-07-06 08:31:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 08:31:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 08:31:34,895 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 08:31:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:32:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:31:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:31:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:31:51.979497+00:00 (in 14.999545 seconds) quotatotalworker stdout | 2023-07-06 08:31:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:31:51 UTC)" (scheduled at 2023-07-06 08:31:36.979497+00:00) quotatotalworker stdout | 2023-07-06 08:31:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:31:36,990 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:31:36,990 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:31:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:31:51,065 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:31:51,339 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:31:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:31:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:32:06.979497+00:00 (in 14.999537 seconds) quotatotalworker stdout | 2023-07-06 08:31:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:32:06 UTC)" (scheduled at 2023-07-06 08:31:51.979497+00:00) quotatotalworker stdout | 2023-07-06 08:31:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:31:51,998 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:31:51,998 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:32:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:32:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:32:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:32:21.979497+00:00 (in 14.999539 seconds) quotatotalworker stdout | 2023-07-06 08:32:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:32:21 UTC)" (scheduled at 2023-07-06 08:32:06.979497+00:00) quotatotalworker stdout | 2023-07-06 08:32:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:32:06,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:32:06,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:32:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:32:21,076 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:32:21,370 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:32:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:32:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:32:36.979497+00:00 (in 14.999576 seconds) quotatotalworker stdout | 2023-07-06 08:32:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:32:36 UTC)" (scheduled at 2023-07-06 08:32:21.979497+00:00) quotatotalworker stdout | 2023-07-06 08:32:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:32:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:32:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:32:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 08:32:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 08:32:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:33:34.887001+00:00 (in 59.999497 seconds) quotaregistrysizeworker stdout | 2023-07-06 08:32:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:33:34 UTC)" (scheduled at 2023-07-06 08:32:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 08:32:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 08:32:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 08:32:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:33:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:32:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:32:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:32:51.979497+00:00 (in 14.999539 seconds) quotatotalworker stdout | 2023-07-06 08:32:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:32:51 UTC)" (scheduled at 2023-07-06 08:32:36.979497+00:00) quotatotalworker stdout | 2023-07-06 08:32:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:32:36,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:32:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:32:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:32:51,100 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:32:51,402 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:32:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:32:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:33:06.979497+00:00 (in 14.999586 seconds) quotatotalworker stdout | 2023-07-06 08:32:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:33:06 UTC)" (scheduled at 2023-07-06 08:32:51.979497+00:00) quotatotalworker stdout | 2023-07-06 08:32:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:32:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:32:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:33:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:33:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:33:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:33:21.979497+00:00 (in 14.999542 seconds) quotatotalworker stdout | 2023-07-06 08:33:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:33:21 UTC)" (scheduled at 2023-07-06 08:33:06.979497+00:00) quotatotalworker stdout | 2023-07-06 08:33:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:33:06,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:33:06,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:33:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:33:21,133 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:33:21,434 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:33:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:33:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:33:36.979497+00:00 (in 14.999560 seconds) quotatotalworker stdout | 2023-07-06 08:33:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:33:36 UTC)" (scheduled at 2023-07-06 08:33:21.979497+00:00) quotatotalworker stdout | 2023-07-06 08:33:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:33:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:33:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:33:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 08:33:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 08:33:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:34:34.887001+00:00 (in 59.999469 seconds) quotaregistrysizeworker stdout | 2023-07-06 08:33:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:34:34 UTC)" (scheduled at 2023-07-06 08:33:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 08:33:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 08:33:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 08:33:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:34:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:33:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:33:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:33:51.979497+00:00 (in 14.999559 seconds) quotatotalworker stdout | 2023-07-06 08:33:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:33:51 UTC)" (scheduled at 2023-07-06 08:33:36.979497+00:00) quotatotalworker stdout | 2023-07-06 08:33:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:33:36,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:33:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:33:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:33:51,165 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:33:51,466 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:33:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:33:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:34:06.979497+00:00 (in 14.999550 seconds) quotatotalworker stdout | 2023-07-06 08:33:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:34:06 UTC)" (scheduled at 2023-07-06 08:33:51.979497+00:00) quotatotalworker stdout | 2023-07-06 08:33:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:33:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:33:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:34:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:34:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:34:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:34:21.979497+00:00 (in 14.999526 seconds) quotatotalworker stdout | 2023-07-06 08:34:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:34:21 UTC)" (scheduled at 2023-07-06 08:34:06.979497+00:00) quotatotalworker stdout | 2023-07-06 08:34:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:34:06,999 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:34:06,999 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:34:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:34:21,167 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:34:21,498 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:34:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:34:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:34:36.979497+00:00 (in 14.999569 seconds) quotatotalworker stdout | 2023-07-06 08:34:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:34:36 UTC)" (scheduled at 2023-07-06 08:34:21.979497+00:00) quotatotalworker stdout | 2023-07-06 08:34:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:34:21,987 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:34:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:34:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 08:34:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 08:34:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:35:34.887001+00:00 (in 59.999529 seconds) quotaregistrysizeworker stdout | 2023-07-06 08:34:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:35:34 UTC)" (scheduled at 2023-07-06 08:34:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 08:34:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 08:34:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 08:34:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:35:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:34:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:34:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:34:51.979497+00:00 (in 14.999551 seconds) quotatotalworker stdout | 2023-07-06 08:34:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:34:51 UTC)" (scheduled at 2023-07-06 08:34:36.979497+00:00) quotatotalworker stdout | 2023-07-06 08:34:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:34:36,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:34:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:34:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:34:51,190 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:34:51,524 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:34:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:34:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:35:06.979497+00:00 (in 14.999585 seconds) quotatotalworker stdout | 2023-07-06 08:34:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:35:06 UTC)" (scheduled at 2023-07-06 08:34:51.979497+00:00) quotatotalworker stdout | 2023-07-06 08:34:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:34:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:34:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:35:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:35:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:35:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:35:21.979497+00:00 (in 14.999618 seconds) quotatotalworker stdout | 2023-07-06 08:35:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:35:21 UTC)" (scheduled at 2023-07-06 08:35:06.979497+00:00) quotatotalworker stdout | 2023-07-06 08:35:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:35:06,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:35:06,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:35:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:35:21,219 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:35:21,556 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:35:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:35:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:35:36.979497+00:00 (in 14.999609 seconds) quotatotalworker stdout | 2023-07-06 08:35:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:35:36 UTC)" (scheduled at 2023-07-06 08:35:21.979497+00:00) quotatotalworker stdout | 2023-07-06 08:35:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:35:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:35:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:35:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 08:35:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 08:35:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:36:34.887001+00:00 (in 59.999521 seconds) quotaregistrysizeworker stdout | 2023-07-06 08:35:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:36:34 UTC)" (scheduled at 2023-07-06 08:35:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 08:35:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 08:35:34,895 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 08:35:34,895 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:36:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:35:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:35:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:35:51.979497+00:00 (in 14.999564 seconds) quotatotalworker stdout | 2023-07-06 08:35:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:35:51 UTC)" (scheduled at 2023-07-06 08:35:36.979497+00:00) quotatotalworker stdout | 2023-07-06 08:35:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:35:36,990 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:35:36,990 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:35:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:35:51,251 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:35:51,589 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:35:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:35:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:36:06.979497+00:00 (in 14.999565 seconds) quotatotalworker stdout | 2023-07-06 08:35:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:36:06 UTC)" (scheduled at 2023-07-06 08:35:51.979497+00:00) quotatotalworker stdout | 2023-07-06 08:35:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:35:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:35:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:36:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:36:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:36:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:36:21.979497+00:00 (in 14.999524 seconds) quotatotalworker stdout | 2023-07-06 08:36:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:36:21 UTC)" (scheduled at 2023-07-06 08:36:06.979497+00:00) quotatotalworker stdout | 2023-07-06 08:36:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:36:06,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:36:06,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:36:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:36:21,276 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:36:21,607 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:36:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:36:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:36:36.979497+00:00 (in 14.999563 seconds) quotatotalworker stdout | 2023-07-06 08:36:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:36:36 UTC)" (scheduled at 2023-07-06 08:36:21.979497+00:00) quotatotalworker stdout | 2023-07-06 08:36:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:36:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:36:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:36:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 08:36:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 08:36:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:37:34.887001+00:00 (in 59.999504 seconds) quotaregistrysizeworker stdout | 2023-07-06 08:36:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:37:34 UTC)" (scheduled at 2023-07-06 08:36:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 08:36:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 08:36:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 08:36:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:37:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:36:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:36:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:36:51.979497+00:00 (in 14.999538 seconds) quotatotalworker stdout | 2023-07-06 08:36:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:36:51 UTC)" (scheduled at 2023-07-06 08:36:36.979497+00:00) quotatotalworker stdout | 2023-07-06 08:36:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:36:36,998 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:36:36,998 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:36:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:36:51,308 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:36:51,639 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:36:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:36:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:37:06.979497+00:00 (in 14.999579 seconds) quotatotalworker stdout | 2023-07-06 08:36:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:37:06 UTC)" (scheduled at 2023-07-06 08:36:51.979497+00:00) quotatotalworker stdout | 2023-07-06 08:36:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:36:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:36:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:37:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:37:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:37:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:37:21.979497+00:00 (in 14.999539 seconds) quotatotalworker stdout | 2023-07-06 08:37:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:37:21 UTC)" (scheduled at 2023-07-06 08:37:06.979497+00:00) quotatotalworker stdout | 2023-07-06 08:37:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:37:06,990 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:37:06,990 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:37:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:37:21,339 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:37:21,652 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:37:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:37:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:37:36.979497+00:00 (in 14.999511 seconds) quotatotalworker stdout | 2023-07-06 08:37:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:37:36 UTC)" (scheduled at 2023-07-06 08:37:21.979497+00:00) quotatotalworker stdout | 2023-07-06 08:37:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:37:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:37:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:37:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 08:37:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 08:37:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:38:34.887001+00:00 (in 59.999529 seconds) quotaregistrysizeworker stdout | 2023-07-06 08:37:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:38:34 UTC)" (scheduled at 2023-07-06 08:37:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 08:37:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 08:37:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 08:37:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:38:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:37:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:37:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:37:51.979497+00:00 (in 14.999527 seconds) quotatotalworker stdout | 2023-07-06 08:37:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:37:51 UTC)" (scheduled at 2023-07-06 08:37:36.979497+00:00) quotatotalworker stdout | 2023-07-06 08:37:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:37:36,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:37:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:37:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:37:51,349 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:37:51,662 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:37:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:37:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:38:06.979497+00:00 (in 14.999538 seconds) quotatotalworker stdout | 2023-07-06 08:37:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:38:06 UTC)" (scheduled at 2023-07-06 08:37:51.979497+00:00) quotatotalworker stdout | 2023-07-06 08:37:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:37:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:37:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:38:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:38:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:38:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:38:21.979497+00:00 (in 14.999532 seconds) quotatotalworker stdout | 2023-07-06 08:38:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:38:21 UTC)" (scheduled at 2023-07-06 08:38:06.979497+00:00) quotatotalworker stdout | 2023-07-06 08:38:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:38:06,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:38:06,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:38:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:38:21,360 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:38:21,694 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:38:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:38:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:38:36.979497+00:00 (in 14.999585 seconds) quotatotalworker stdout | 2023-07-06 08:38:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:38:36 UTC)" (scheduled at 2023-07-06 08:38:21.979497+00:00) quotatotalworker stdout | 2023-07-06 08:38:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:38:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:38:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:38:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 08:38:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 08:38:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:39:34.887001+00:00 (in 59.999495 seconds) quotaregistrysizeworker stdout | 2023-07-06 08:38:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:39:34 UTC)" (scheduled at 2023-07-06 08:38:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 08:38:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 08:38:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 08:38:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:39:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:38:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:38:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:38:51.979497+00:00 (in 14.999558 seconds) quotatotalworker stdout | 2023-07-06 08:38:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:38:51 UTC)" (scheduled at 2023-07-06 08:38:36.979497+00:00) quotatotalworker stdout | 2023-07-06 08:38:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:38:36,996 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:38:36,997 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:38:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:38:51,391 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:38:51,710 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:38:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:38:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:39:06.979497+00:00 (in 14.999584 seconds) quotatotalworker stdout | 2023-07-06 08:38:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:39:06 UTC)" (scheduled at 2023-07-06 08:38:51.979497+00:00) quotatotalworker stdout | 2023-07-06 08:38:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:38:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:38:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:39:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:39:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:39:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:39:21.979497+00:00 (in 14.999568 seconds) quotatotalworker stdout | 2023-07-06 08:39:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:39:21 UTC)" (scheduled at 2023-07-06 08:39:06.979497+00:00) quotatotalworker stdout | 2023-07-06 08:39:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:39:06,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:39:06,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:39:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:39:21,408 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:39:21,725 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:39:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:39:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:39:36.979497+00:00 (in 14.999571 seconds) quotatotalworker stdout | 2023-07-06 08:39:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:39:36 UTC)" (scheduled at 2023-07-06 08:39:21.979497+00:00) quotatotalworker stdout | 2023-07-06 08:39:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:39:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:39:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:39:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 08:39:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 08:39:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:40:34.887001+00:00 (in 59.999537 seconds) quotaregistrysizeworker stdout | 2023-07-06 08:39:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:40:34 UTC)" (scheduled at 2023-07-06 08:39:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 08:39:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 08:39:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 08:39:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:40:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:39:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:39:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:39:51.979497+00:00 (in 14.999531 seconds) quotatotalworker stdout | 2023-07-06 08:39:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:39:51 UTC)" (scheduled at 2023-07-06 08:39:36.979497+00:00) quotatotalworker stdout | 2023-07-06 08:39:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:39:36,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:39:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:39:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:39:51,439 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:39:51,757 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:39:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:39:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:40:06.979497+00:00 (in 14.999564 seconds) quotatotalworker stdout | 2023-07-06 08:39:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:40:06 UTC)" (scheduled at 2023-07-06 08:39:51.979497+00:00) quotatotalworker stdout | 2023-07-06 08:39:51,980 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:39:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:39:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:40:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:40:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:40:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:40:21.979497+00:00 (in 14.999553 seconds) quotatotalworker stdout | 2023-07-06 08:40:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:40:21 UTC)" (scheduled at 2023-07-06 08:40:06.979497+00:00) quotatotalworker stdout | 2023-07-06 08:40:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:40:06,990 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:40:06,990 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:40:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:40:21,471 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:40:21,778 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:40:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:40:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:40:36.979497+00:00 (in 14.999591 seconds) quotatotalworker stdout | 2023-07-06 08:40:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:40:36 UTC)" (scheduled at 2023-07-06 08:40:21.979497+00:00) quotatotalworker stdout | 2023-07-06 08:40:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:40:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:40:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:40:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 08:40:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 08:40:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:41:34.887001+00:00 (in 59.999518 seconds) quotaregistrysizeworker stdout | 2023-07-06 08:40:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:41:34 UTC)" (scheduled at 2023-07-06 08:40:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 08:40:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 08:40:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 08:40:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:41:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:40:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:40:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:40:51.979497+00:00 (in 14.999573 seconds) quotatotalworker stdout | 2023-07-06 08:40:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:40:51 UTC)" (scheduled at 2023-07-06 08:40:36.979497+00:00) quotatotalworker stdout | 2023-07-06 08:40:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:40:36,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:40:36,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:40:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:40:51,494 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:40:51,810 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:40:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:40:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:41:06.979497+00:00 (in 14.999592 seconds) quotatotalworker stdout | 2023-07-06 08:40:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:41:06 UTC)" (scheduled at 2023-07-06 08:40:51.979497+00:00) quotatotalworker stdout | 2023-07-06 08:40:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:40:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:40:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:41:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:41:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:41:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:41:21.979497+00:00 (in 14.999542 seconds) quotatotalworker stdout | 2023-07-06 08:41:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:41:21 UTC)" (scheduled at 2023-07-06 08:41:06.979497+00:00) quotatotalworker stdout | 2023-07-06 08:41:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:41:06,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:41:06,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:41:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:41:21,512 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:41:21,829 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:41:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:41:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:41:36.979497+00:00 (in 14.999559 seconds) quotatotalworker stdout | 2023-07-06 08:41:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:41:36 UTC)" (scheduled at 2023-07-06 08:41:21.979497+00:00) quotatotalworker stdout | 2023-07-06 08:41:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:41:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:41:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:41:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 08:41:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 08:41:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:42:34.887001+00:00 (in 59.999514 seconds) quotaregistrysizeworker stdout | 2023-07-06 08:41:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:42:34 UTC)" (scheduled at 2023-07-06 08:41:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 08:41:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 08:41:34,895 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 08:41:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:42:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:41:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:41:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:41:51.979497+00:00 (in 14.999541 seconds) quotatotalworker stdout | 2023-07-06 08:41:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:41:51 UTC)" (scheduled at 2023-07-06 08:41:36.979497+00:00) quotatotalworker stdout | 2023-07-06 08:41:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:41:36,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:41:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:41:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:41:51,544 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:41:51,861 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:41:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:41:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:42:06.979497+00:00 (in 14.999559 seconds) quotatotalworker stdout | 2023-07-06 08:41:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:42:06 UTC)" (scheduled at 2023-07-06 08:41:51.979497+00:00) quotatotalworker stdout | 2023-07-06 08:41:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:41:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:41:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:42:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:42:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:42:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:42:21.979497+00:00 (in 14.999549 seconds) quotatotalworker stdout | 2023-07-06 08:42:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:42:21 UTC)" (scheduled at 2023-07-06 08:42:06.979497+00:00) quotatotalworker stdout | 2023-07-06 08:42:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:42:06,990 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:42:06,990 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:42:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:42:21,575 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:42:21,893 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:42:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:42:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:42:36.979497+00:00 (in 14.999531 seconds) quotatotalworker stdout | 2023-07-06 08:42:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:42:36 UTC)" (scheduled at 2023-07-06 08:42:21.979497+00:00) quotatotalworker stdout | 2023-07-06 08:42:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:42:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:42:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:42:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 08:42:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 08:42:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:43:34.887001+00:00 (in 59.999435 seconds) quotaregistrysizeworker stdout | 2023-07-06 08:42:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:43:34 UTC)" (scheduled at 2023-07-06 08:42:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 08:42:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 08:42:34,895 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 08:42:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:43:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:42:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:42:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:42:51.979497+00:00 (in 14.999517 seconds) quotatotalworker stdout | 2023-07-06 08:42:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:42:51 UTC)" (scheduled at 2023-07-06 08:42:36.979497+00:00) quotatotalworker stdout | 2023-07-06 08:42:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:42:36,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:42:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:42:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:42:51,606 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:42:51,925 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:42:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:42:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:43:06.979497+00:00 (in 14.999572 seconds) quotatotalworker stdout | 2023-07-06 08:42:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:43:06 UTC)" (scheduled at 2023-07-06 08:42:51.979497+00:00) quotatotalworker stdout | 2023-07-06 08:42:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:42:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:42:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:43:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:43:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:43:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:43:21.979497+00:00 (in 14.999543 seconds) quotatotalworker stdout | 2023-07-06 08:43:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:43:21 UTC)" (scheduled at 2023-07-06 08:43:06.979497+00:00) quotatotalworker stdout | 2023-07-06 08:43:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:43:06,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:43:06,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:43:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:43:21,638 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:43:21,958 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:43:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:43:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:43:36.979497+00:00 (in 14.999579 seconds) quotatotalworker stdout | 2023-07-06 08:43:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:43:36 UTC)" (scheduled at 2023-07-06 08:43:21.979497+00:00) quotatotalworker stdout | 2023-07-06 08:43:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:43:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:43:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:43:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 08:43:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 08:43:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:44:34.887001+00:00 (in 59.999555 seconds) quotaregistrysizeworker stdout | 2023-07-06 08:43:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:44:34 UTC)" (scheduled at 2023-07-06 08:43:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 08:43:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 08:43:34,895 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 08:43:34,895 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:44:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:43:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:43:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:43:51.979497+00:00 (in 14.999540 seconds) quotatotalworker stdout | 2023-07-06 08:43:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:43:51 UTC)" (scheduled at 2023-07-06 08:43:36.979497+00:00) quotatotalworker stdout | 2023-07-06 08:43:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:43:36,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:43:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:43:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:43:51,670 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:43:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:43:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:44:06.979497+00:00 (in 14.999578 seconds) quotatotalworker stdout | 2023-07-06 08:43:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:44:06 UTC)" (scheduled at 2023-07-06 08:43:51.979497+00:00) quotatotalworker stdout | 2023-07-06 08:43:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:43:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:43:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:44:06 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 08:43:51,990 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:44:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:44:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:44:21.979497+00:00 (in 14.999570 seconds) quotatotalworker stdout | 2023-07-06 08:44:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:44:21 UTC)" (scheduled at 2023-07-06 08:44:06.979497+00:00) quotatotalworker stdout | 2023-07-06 08:44:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:44:06,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:44:06,990 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:44:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:44:21,701 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:44:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:44:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:44:36.979497+00:00 (in 14.999602 seconds) quotatotalworker stdout | 2023-07-06 08:44:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:44:36 UTC)" (scheduled at 2023-07-06 08:44:21.979497+00:00) quotatotalworker stdout | 2023-07-06 08:44:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:44:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:44:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:44:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 08:44:22,022 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:44:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 08:44:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:45:34.887001+00:00 (in 59.999557 seconds) quotaregistrysizeworker stdout | 2023-07-06 08:44:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:45:34 UTC)" (scheduled at 2023-07-06 08:44:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 08:44:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 08:44:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 08:44:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:45:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:44:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:44:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:44:51.979497+00:00 (in 14.999552 seconds) quotatotalworker stdout | 2023-07-06 08:44:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:44:51 UTC)" (scheduled at 2023-07-06 08:44:36.979497+00:00) quotatotalworker stdout | 2023-07-06 08:44:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:44:36,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:44:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:44:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:44:51,719 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:44:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:44:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:45:06.979497+00:00 (in 14.999567 seconds) quotatotalworker stdout | 2023-07-06 08:44:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:45:06 UTC)" (scheduled at 2023-07-06 08:44:51.979497+00:00) quotatotalworker stdout | 2023-07-06 08:44:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:44:51,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:44:51,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:45:06 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 08:44:52,054 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:45:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:45:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:45:21.979497+00:00 (in 14.999574 seconds) quotatotalworker stdout | 2023-07-06 08:45:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:45:21 UTC)" (scheduled at 2023-07-06 08:45:06.979497+00:00) quotatotalworker stdout | 2023-07-06 08:45:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:45:06,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:45:06,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:45:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:45:21,730 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:45:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:45:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:45:36.979497+00:00 (in 14.999617 seconds) quotatotalworker stdout | 2023-07-06 08:45:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:45:36 UTC)" (scheduled at 2023-07-06 08:45:21.979497+00:00) quotatotalworker stdout | 2023-07-06 08:45:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:45:21,987 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:45:21,987 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:45:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 08:45:22,086 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:45:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 08:45:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:46:34.887001+00:00 (in 59.999578 seconds) quotaregistrysizeworker stdout | 2023-07-06 08:45:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:46:34 UTC)" (scheduled at 2023-07-06 08:45:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 08:45:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 08:45:34,895 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 08:45:34,895 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:46:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:45:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:45:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:45:51.979497+00:00 (in 14.999532 seconds) quotatotalworker stdout | 2023-07-06 08:45:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:45:51 UTC)" (scheduled at 2023-07-06 08:45:36.979497+00:00) quotatotalworker stdout | 2023-07-06 08:45:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:45:36,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:45:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:45:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:45:51,752 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:45:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:45:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:46:06.979497+00:00 (in 14.999594 seconds) quotatotalworker stdout | 2023-07-06 08:45:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:46:06 UTC)" (scheduled at 2023-07-06 08:45:51.979497+00:00) quotatotalworker stdout | 2023-07-06 08:45:51,980 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:45:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:45:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:46:06 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 08:45:52,118 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:46:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:46:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:46:21.979497+00:00 (in 14.999574 seconds) quotatotalworker stdout | 2023-07-06 08:46:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:46:21 UTC)" (scheduled at 2023-07-06 08:46:06.979497+00:00) quotatotalworker stdout | 2023-07-06 08:46:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:46:06,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:46:06,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:46:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:46:21,781 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:46:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:46:21,979 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:46:36.979497+00:00 (in 14.999635 seconds) quotatotalworker stdout | 2023-07-06 08:46:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:46:36 UTC)" (scheduled at 2023-07-06 08:46:21.979497+00:00) quotatotalworker stdout | 2023-07-06 08:46:21,980 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:46:21,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:46:21,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:46:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 08:46:22,150 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:46:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 08:46:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:47:34.887001+00:00 (in 59.999585 seconds) quotaregistrysizeworker stdout | 2023-07-06 08:46:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:47:34 UTC)" (scheduled at 2023-07-06 08:46:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 08:46:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 08:46:34,895 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 08:46:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:47:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:46:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:46:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:46:51.979497+00:00 (in 14.999573 seconds) quotatotalworker stdout | 2023-07-06 08:46:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:46:51 UTC)" (scheduled at 2023-07-06 08:46:36.979497+00:00) quotatotalworker stdout | 2023-07-06 08:46:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:46:36,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:46:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:46:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:46:51,793 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:46:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:46:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:47:06.979497+00:00 (in 14.999608 seconds) quotatotalworker stdout | 2023-07-06 08:46:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:47:06 UTC)" (scheduled at 2023-07-06 08:46:51.979497+00:00) quotatotalworker stdout | 2023-07-06 08:46:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:46:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:46:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:47:06 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 08:46:52,182 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:47:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:47:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:47:21.979497+00:00 (in 14.999581 seconds) quotatotalworker stdout | 2023-07-06 08:47:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:47:21 UTC)" (scheduled at 2023-07-06 08:47:06.979497+00:00) quotatotalworker stdout | 2023-07-06 08:47:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:47:06,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:47:06,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:47:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:47:21,825 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:47:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:47:21,979 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:47:36.979497+00:00 (in 14.999639 seconds) quotatotalworker stdout | 2023-07-06 08:47:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:47:36 UTC)" (scheduled at 2023-07-06 08:47:21.979497+00:00) quotatotalworker stdout | 2023-07-06 08:47:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:47:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:47:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:47:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 08:47:22,196 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:47:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 08:47:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:48:34.887001+00:00 (in 59.999568 seconds) quotaregistrysizeworker stdout | 2023-07-06 08:47:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:48:34 UTC)" (scheduled at 2023-07-06 08:47:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 08:47:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 08:47:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 08:47:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:48:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:47:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:47:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:47:51.979497+00:00 (in 14.999569 seconds) quotatotalworker stdout | 2023-07-06 08:47:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:47:51 UTC)" (scheduled at 2023-07-06 08:47:36.979497+00:00) quotatotalworker stdout | 2023-07-06 08:47:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:47:36,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:47:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:47:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:47:51,852 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:47:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:47:51,979 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:48:06.979497+00:00 (in 14.999639 seconds) quotatotalworker stdout | 2023-07-06 08:47:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:48:06 UTC)" (scheduled at 2023-07-06 08:47:51.979497+00:00) quotatotalworker stdout | 2023-07-06 08:47:51,980 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:47:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:47:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:48:06 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 08:47:52,215 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:48:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:48:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:48:21.979497+00:00 (in 14.999601 seconds) quotatotalworker stdout | 2023-07-06 08:48:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:48:21 UTC)" (scheduled at 2023-07-06 08:48:06.979497+00:00) quotatotalworker stdout | 2023-07-06 08:48:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:48:06,990 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:48:06,990 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:48:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:48:21,870 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:48:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:48:21,979 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:48:36.979497+00:00 (in 14.999631 seconds) quotatotalworker stdout | 2023-07-06 08:48:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:48:36 UTC)" (scheduled at 2023-07-06 08:48:21.979497+00:00) quotatotalworker stdout | 2023-07-06 08:48:21,980 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:48:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:48:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:48:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 08:48:22,231 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:48:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 08:48:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:49:34.887001+00:00 (in 59.999531 seconds) quotaregistrysizeworker stdout | 2023-07-06 08:48:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:49:34 UTC)" (scheduled at 2023-07-06 08:48:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 08:48:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 08:48:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 08:48:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:49:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:48:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:48:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:48:51.979497+00:00 (in 14.999558 seconds) quotatotalworker stdout | 2023-07-06 08:48:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:48:51 UTC)" (scheduled at 2023-07-06 08:48:36.979497+00:00) quotatotalworker stdout | 2023-07-06 08:48:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:48:36,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:48:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:48:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:48:51,878 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:48:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:48:51,979 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:49:06.979497+00:00 (in 14.999652 seconds) quotatotalworker stdout | 2023-07-06 08:48:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:49:06 UTC)" (scheduled at 2023-07-06 08:48:51.979497+00:00) quotatotalworker stdout | 2023-07-06 08:48:51,980 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:48:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:48:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:49:06 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 08:48:52,252 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:49:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:49:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:49:21.979497+00:00 (in 14.999558 seconds) quotatotalworker stdout | 2023-07-06 08:49:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:49:21 UTC)" (scheduled at 2023-07-06 08:49:06.979497+00:00) quotatotalworker stdout | 2023-07-06 08:49:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:49:06,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:49:06,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:49:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:49:21,910 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:49:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:49:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:49:36.979497+00:00 (in 14.999632 seconds) quotatotalworker stdout | 2023-07-06 08:49:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:49:36 UTC)" (scheduled at 2023-07-06 08:49:21.979497+00:00) quotatotalworker stdout | 2023-07-06 08:49:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:49:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:49:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:49:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 08:49:22,272 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:49:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 08:49:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:50:34.887001+00:00 (in 59.999510 seconds) quotaregistrysizeworker stdout | 2023-07-06 08:49:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:50:34 UTC)" (scheduled at 2023-07-06 08:49:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 08:49:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 08:49:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 08:49:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:50:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:49:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:49:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:49:51.979497+00:00 (in 14.999564 seconds) quotatotalworker stdout | 2023-07-06 08:49:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:49:51 UTC)" (scheduled at 2023-07-06 08:49:36.979497+00:00) quotatotalworker stdout | 2023-07-06 08:49:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:49:36,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:49:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:49:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:49:51,938 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:49:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:49:51,979 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:50:06.979497+00:00 (in 14.999631 seconds) quotatotalworker stdout | 2023-07-06 08:49:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:50:06 UTC)" (scheduled at 2023-07-06 08:49:51.979497+00:00) quotatotalworker stdout | 2023-07-06 08:49:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:49:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:49:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:50:06 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 08:49:52,303 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:50:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:50:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:50:21.979497+00:00 (in 14.999617 seconds) quotatotalworker stdout | 2023-07-06 08:50:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:50:21 UTC)" (scheduled at 2023-07-06 08:50:06.979497+00:00) quotatotalworker stdout | 2023-07-06 08:50:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:50:06,990 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:50:06,990 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:50:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:50:21,958 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:50:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:50:21,979 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:50:36.979497+00:00 (in 14.999676 seconds) quotatotalworker stdout | 2023-07-06 08:50:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:50:36 UTC)" (scheduled at 2023-07-06 08:50:21.979497+00:00) quotatotalworker stdout | 2023-07-06 08:50:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:50:21,987 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:50:21,987 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:50:36 UTC)" executed successfully quotaregistrysizeworker stdout | 2023-07-06 08:50:22,308 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:50:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 08:50:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:51:34.887001+00:00 (in 59.999505 seconds) quotaregistrysizeworker stdout | 2023-07-06 08:50:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:51:34 UTC)" (scheduled at 2023-07-06 08:50:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 08:50:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 08:50:34,895 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 08:50:34,895 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:51:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:50:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:50:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:50:51.979497+00:00 (in 14.999606 seconds) quotatotalworker stdout | 2023-07-06 08:50:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:50:51 UTC)" (scheduled at 2023-07-06 08:50:36.979497+00:00) quotatotalworker stdout | 2023-07-06 08:50:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:50:36,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:50:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:50:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:50:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:50:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:51:06.979497+00:00 (in 14.999569 seconds) quotatotalworker stdout | 2023-07-06 08:50:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:51:06 UTC)" (scheduled at 2023-07-06 08:50:51.979497+00:00) quotatotalworker stdout | 2023-07-06 08:50:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:50:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:50:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:51:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:50:51,989 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:50:52,340 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:51:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:51:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:51:21.979497+00:00 (in 14.999626 seconds) quotatotalworker stdout | 2023-07-06 08:51:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:51:21 UTC)" (scheduled at 2023-07-06 08:51:06.979497+00:00) quotatotalworker stdout | 2023-07-06 08:51:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:51:06,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:51:06,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:51:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:51:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:51:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:51:36.979497+00:00 (in 14.999589 seconds) quotatotalworker stdout | 2023-07-06 08:51:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:51:36 UTC)" (scheduled at 2023-07-06 08:51:21.979497+00:00) quotatotalworker stdout | 2023-07-06 08:51:21,980 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:51:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:51:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:51:36 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:51:22,007 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:51:22,342 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:51:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 08:51:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:52:34.887001+00:00 (in 59.999479 seconds) quotaregistrysizeworker stdout | 2023-07-06 08:51:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:52:34 UTC)" (scheduled at 2023-07-06 08:51:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 08:51:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 08:51:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 08:51:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:52:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:51:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:51:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:51:51.979497+00:00 (in 14.999567 seconds) quotatotalworker stdout | 2023-07-06 08:51:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:51:51 UTC)" (scheduled at 2023-07-06 08:51:36.979497+00:00) quotatotalworker stdout | 2023-07-06 08:51:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:51:36,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:51:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:51:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:51:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:51:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:52:06.979497+00:00 (in 14.999572 seconds) quotatotalworker stdout | 2023-07-06 08:51:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:52:06 UTC)" (scheduled at 2023-07-06 08:51:51.979497+00:00) quotatotalworker stdout | 2023-07-06 08:51:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:51:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:51:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:52:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:51:52,019 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:51:52,345 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:52:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:52:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:52:21.979497+00:00 (in 14.999575 seconds) quotatotalworker stdout | 2023-07-06 08:52:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:52:21 UTC)" (scheduled at 2023-07-06 08:52:06.979497+00:00) quotatotalworker stdout | 2023-07-06 08:52:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:52:06,990 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:52:06,990 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:52:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:52:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:52:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:52:36.979497+00:00 (in 14.999552 seconds) quotatotalworker stdout | 2023-07-06 08:52:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:52:36 UTC)" (scheduled at 2023-07-06 08:52:21.979497+00:00) quotatotalworker stdout | 2023-07-06 08:52:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:52:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:52:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:52:36 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:52:22,051 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:52:22,377 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:52:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 08:52:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:53:34.887001+00:00 (in 59.999499 seconds) quotaregistrysizeworker stdout | 2023-07-06 08:52:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:53:34 UTC)" (scheduled at 2023-07-06 08:52:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 08:52:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 08:52:34,895 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 08:52:34,895 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:53:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:52:36,980 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:52:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:52:51.979497+00:00 (in 14.999172 seconds) quotatotalworker stdout | 2023-07-06 08:52:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:52:51 UTC)" (scheduled at 2023-07-06 08:52:36.979497+00:00) quotatotalworker stdout | 2023-07-06 08:52:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:52:36,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:52:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:52:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:52:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:52:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:53:06.979497+00:00 (in 14.999529 seconds) quotatotalworker stdout | 2023-07-06 08:52:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:53:06 UTC)" (scheduled at 2023-07-06 08:52:51.979497+00:00) quotatotalworker stdout | 2023-07-06 08:52:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:52:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:52:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:53:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:52:52,084 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:52:52,409 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:53:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:53:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:53:21.979497+00:00 (in 14.999530 seconds) quotatotalworker stdout | 2023-07-06 08:53:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:53:21 UTC)" (scheduled at 2023-07-06 08:53:06.979497+00:00) quotatotalworker stdout | 2023-07-06 08:53:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:53:06,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:53:06,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:53:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:53:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:53:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:53:36.979497+00:00 (in 14.999568 seconds) quotatotalworker stdout | 2023-07-06 08:53:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:53:36 UTC)" (scheduled at 2023-07-06 08:53:21.979497+00:00) quotatotalworker stdout | 2023-07-06 08:53:21,980 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:53:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:53:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:53:36 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:53:22,116 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:53:22,441 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:53:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 08:53:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:54:34.887001+00:00 (in 59.999528 seconds) quotaregistrysizeworker stdout | 2023-07-06 08:53:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:54:34 UTC)" (scheduled at 2023-07-06 08:53:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 08:53:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 08:53:34,895 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 08:53:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:54:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:53:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:53:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:53:51.979497+00:00 (in 14.999568 seconds) quotatotalworker stdout | 2023-07-06 08:53:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:53:51 UTC)" (scheduled at 2023-07-06 08:53:36.979497+00:00) quotatotalworker stdout | 2023-07-06 08:53:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:53:36,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:53:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:53:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:53:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:53:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:54:06.979497+00:00 (in 14.999601 seconds) quotatotalworker stdout | 2023-07-06 08:53:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:54:06 UTC)" (scheduled at 2023-07-06 08:53:51.979497+00:00) quotatotalworker stdout | 2023-07-06 08:53:51,980 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:53:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:53:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:54:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:53:52,136 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:53:52,468 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:54:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:54:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:54:21.979497+00:00 (in 14.999551 seconds) quotatotalworker stdout | 2023-07-06 08:54:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:54:21 UTC)" (scheduled at 2023-07-06 08:54:06.979497+00:00) quotatotalworker stdout | 2023-07-06 08:54:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:54:06,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:54:06,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:54:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:54:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:54:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:54:36.979497+00:00 (in 14.999576 seconds) quotatotalworker stdout | 2023-07-06 08:54:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:54:36 UTC)" (scheduled at 2023-07-06 08:54:21.979497+00:00) quotatotalworker stdout | 2023-07-06 08:54:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:54:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:54:21,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:54:36 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:54:22,151 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:54:22,500 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:54:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 08:54:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:55:34.887001+00:00 (in 59.999534 seconds) quotaregistrysizeworker stdout | 2023-07-06 08:54:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:55:34 UTC)" (scheduled at 2023-07-06 08:54:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 08:54:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 08:54:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 08:54:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:55:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:54:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:54:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:54:51.979497+00:00 (in 14.999571 seconds) quotatotalworker stdout | 2023-07-06 08:54:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:54:51 UTC)" (scheduled at 2023-07-06 08:54:36.979497+00:00) quotatotalworker stdout | 2023-07-06 08:54:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:54:36,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:54:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:54:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:54:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:54:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:55:06.979497+00:00 (in 14.999593 seconds) quotatotalworker stdout | 2023-07-06 08:54:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:55:06 UTC)" (scheduled at 2023-07-06 08:54:51.979497+00:00) quotatotalworker stdout | 2023-07-06 08:54:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:54:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:54:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:55:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:54:52,157 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:54:52,532 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:55:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:55:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:55:21.979497+00:00 (in 14.999561 seconds) quotatotalworker stdout | 2023-07-06 08:55:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:55:21 UTC)" (scheduled at 2023-07-06 08:55:06.979497+00:00) quotatotalworker stdout | 2023-07-06 08:55:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:55:06,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:55:06,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:55:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:55:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:55:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:55:36.979497+00:00 (in 14.999580 seconds) quotatotalworker stdout | 2023-07-06 08:55:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:55:36 UTC)" (scheduled at 2023-07-06 08:55:21.979497+00:00) quotatotalworker stdout | 2023-07-06 08:55:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:55:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:55:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:55:36 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:55:22,183 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:55:22,546 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:55:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 08:55:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:56:34.887001+00:00 (in 59.999532 seconds) quotaregistrysizeworker stdout | 2023-07-06 08:55:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:56:34 UTC)" (scheduled at 2023-07-06 08:55:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 08:55:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 08:55:34,895 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 08:55:34,895 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:56:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:55:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:55:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:55:51.979497+00:00 (in 14.999569 seconds) quotatotalworker stdout | 2023-07-06 08:55:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:55:51 UTC)" (scheduled at 2023-07-06 08:55:36.979497+00:00) quotatotalworker stdout | 2023-07-06 08:55:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:55:36,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:55:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:55:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:55:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:55:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:56:06.979497+00:00 (in 14.999595 seconds) quotatotalworker stdout | 2023-07-06 08:55:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:56:06 UTC)" (scheduled at 2023-07-06 08:55:51.979497+00:00) quotatotalworker stdout | 2023-07-06 08:55:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:55:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:55:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:56:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:55:52,191 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:55:52,577 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:56:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:56:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:56:21.979497+00:00 (in 14.999575 seconds) quotatotalworker stdout | 2023-07-06 08:56:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:56:21 UTC)" (scheduled at 2023-07-06 08:56:06.979497+00:00) quotatotalworker stdout | 2023-07-06 08:56:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:56:06,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:56:06,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:56:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:56:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:56:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:56:36.979497+00:00 (in 14.999555 seconds) quotatotalworker stdout | 2023-07-06 08:56:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:56:36 UTC)" (scheduled at 2023-07-06 08:56:21.979497+00:00) quotatotalworker stdout | 2023-07-06 08:56:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:56:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:56:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:56:36 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:56:22,215 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:56:22,587 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:56:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 08:56:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:57:34.887001+00:00 (in 59.999502 seconds) quotaregistrysizeworker stdout | 2023-07-06 08:56:34,888 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:57:34 UTC)" (scheduled at 2023-07-06 08:56:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 08:56:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 08:56:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 08:56:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:57:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:56:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:56:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:56:51.979497+00:00 (in 14.999563 seconds) quotatotalworker stdout | 2023-07-06 08:56:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:56:51 UTC)" (scheduled at 2023-07-06 08:56:36.979497+00:00) quotatotalworker stdout | 2023-07-06 08:56:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:56:36,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:56:36,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:56:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:56:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:56:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:57:06.979497+00:00 (in 14.999575 seconds) quotatotalworker stdout | 2023-07-06 08:56:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:57:06 UTC)" (scheduled at 2023-07-06 08:56:51.979497+00:00) quotatotalworker stdout | 2023-07-06 08:56:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:56:51,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:56:51,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:57:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:56:52,246 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:56:52,619 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:57:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:57:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:57:21.979497+00:00 (in 14.999499 seconds) quotatotalworker stdout | 2023-07-06 08:57:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:57:21 UTC)" (scheduled at 2023-07-06 08:57:06.979497+00:00) quotatotalworker stdout | 2023-07-06 08:57:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:57:06,990 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:57:06,990 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:57:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:57:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:57:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:57:36.979497+00:00 (in 14.999565 seconds) quotatotalworker stdout | 2023-07-06 08:57:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:57:36 UTC)" (scheduled at 2023-07-06 08:57:21.979497+00:00) quotatotalworker stdout | 2023-07-06 08:57:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:57:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:57:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:57:36 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:57:22,273 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:57:22,633 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:57:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 08:57:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:58:34.887001+00:00 (in 59.999512 seconds) quotaregistrysizeworker stdout | 2023-07-06 08:57:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:58:34 UTC)" (scheduled at 2023-07-06 08:57:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 08:57:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 08:57:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 08:57:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:58:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:57:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:57:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:57:51.979497+00:00 (in 14.999534 seconds) quotatotalworker stdout | 2023-07-06 08:57:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:57:51 UTC)" (scheduled at 2023-07-06 08:57:36.979497+00:00) quotatotalworker stdout | 2023-07-06 08:57:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:57:36,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:57:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:57:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:57:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:57:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:58:06.979497+00:00 (in 14.999567 seconds) quotatotalworker stdout | 2023-07-06 08:57:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:58:06 UTC)" (scheduled at 2023-07-06 08:57:51.979497+00:00) quotatotalworker stdout | 2023-07-06 08:57:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:57:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:57:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:58:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:57:52,305 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:57:52,636 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:58:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:58:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:58:21.979497+00:00 (in 14.999557 seconds) quotatotalworker stdout | 2023-07-06 08:58:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:58:21 UTC)" (scheduled at 2023-07-06 08:58:06.979497+00:00) quotatotalworker stdout | 2023-07-06 08:58:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:58:06,990 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:58:06,990 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:58:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:58:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:58:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:58:36.979497+00:00 (in 14.999575 seconds) quotatotalworker stdout | 2023-07-06 08:58:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:58:36 UTC)" (scheduled at 2023-07-06 08:58:21.979497+00:00) quotatotalworker stdout | 2023-07-06 08:58:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:58:21,999 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:58:21,999 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:58:36 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:58:22,328 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:58:22,668 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:58:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 08:58:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:59:34.887001+00:00 (in 59.999495 seconds) quotaregistrysizeworker stdout | 2023-07-06 08:58:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:59:34 UTC)" (scheduled at 2023-07-06 08:58:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 08:58:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 08:58:34,895 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 08:58:34,895 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 08:59:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:58:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:58:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:58:51.979497+00:00 (in 14.999561 seconds) quotatotalworker stdout | 2023-07-06 08:58:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:58:51 UTC)" (scheduled at 2023-07-06 08:58:36.979497+00:00) quotatotalworker stdout | 2023-07-06 08:58:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:58:36,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:58:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:58:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:58:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:58:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:59:06.979497+00:00 (in 14.999569 seconds) quotatotalworker stdout | 2023-07-06 08:58:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:59:06 UTC)" (scheduled at 2023-07-06 08:58:51.979497+00:00) quotatotalworker stdout | 2023-07-06 08:58:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:58:51,987 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:58:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:59:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:58:52,348 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:58:52,701 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 08:59:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:59:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:59:21.979497+00:00 (in 14.999555 seconds) quotatotalworker stdout | 2023-07-06 08:59:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:59:21 UTC)" (scheduled at 2023-07-06 08:59:06.979497+00:00) quotatotalworker stdout | 2023-07-06 08:59:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:59:06,990 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:59:06,990 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:59:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:59:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:59:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:59:36.979497+00:00 (in 14.999531 seconds) quotatotalworker stdout | 2023-07-06 08:59:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:59:36 UTC)" (scheduled at 2023-07-06 08:59:21.979497+00:00) quotatotalworker stdout | 2023-07-06 08:59:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:59:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:59:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:59:36 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:59:22,362 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:59:22,732 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:59:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 08:59:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 09:00:34.887001+00:00 (in 59.999583 seconds) quotaregistrysizeworker stdout | 2023-07-06 08:59:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 09:00:34 UTC)" (scheduled at 2023-07-06 08:59:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 08:59:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 08:59:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 08:59:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 09:00:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:59:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:59:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 08:59:51.979497+00:00 (in 14.999503 seconds) quotatotalworker stdout | 2023-07-06 08:59:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:59:51 UTC)" (scheduled at 2023-07-06 08:59:36.979497+00:00) quotatotalworker stdout | 2023-07-06 08:59:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:59:36,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:59:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 08:59:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:59:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 08:59:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 09:00:06.979497+00:00 (in 14.999571 seconds) quotatotalworker stdout | 2023-07-06 08:59:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:00:06 UTC)" (scheduled at 2023-07-06 08:59:51.979497+00:00) quotatotalworker stdout | 2023-07-06 08:59:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 08:59:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 08:59:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:00:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 08:59:52,394 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 08:59:52,764 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 09:00:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 09:00:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 09:00:21.979497+00:00 (in 14.999521 seconds) quotatotalworker stdout | 2023-07-06 09:00:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:00:21 UTC)" (scheduled at 2023-07-06 09:00:06.979497+00:00) quotatotalworker stdout | 2023-07-06 09:00:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 09:00:06,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 09:00:06,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:00:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 09:00:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 09:00:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 09:00:36.979497+00:00 (in 14.999530 seconds) quotatotalworker stdout | 2023-07-06 09:00:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:00:36 UTC)" (scheduled at 2023-07-06 09:00:21.979497+00:00) quotatotalworker stdout | 2023-07-06 09:00:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 09:00:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 09:00:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:00:36 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 09:00:22,420 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 09:00:22,797 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 09:00:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 09:00:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 09:01:34.887001+00:00 (in 59.999487 seconds) quotaregistrysizeworker stdout | 2023-07-06 09:00:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 09:01:34 UTC)" (scheduled at 2023-07-06 09:00:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 09:00:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 09:00:34,895 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 09:00:34,895 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 09:01:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 09:00:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 09:00:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 09:00:51.979497+00:00 (in 14.999558 seconds) quotatotalworker stdout | 2023-07-06 09:00:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:00:51 UTC)" (scheduled at 2023-07-06 09:00:36.979497+00:00) quotatotalworker stdout | 2023-07-06 09:00:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 09:00:37,000 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 09:00:37,000 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:00:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 09:00:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 09:00:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 09:01:06.979497+00:00 (in 14.999569 seconds) quotatotalworker stdout | 2023-07-06 09:00:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:01:06 UTC)" (scheduled at 2023-07-06 09:00:51.979497+00:00) quotatotalworker stdout | 2023-07-06 09:00:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 09:00:51,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 09:00:51,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:01:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 09:00:52,437 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 09:00:52,829 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 09:01:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 09:01:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 09:01:21.979497+00:00 (in 14.999608 seconds) quotatotalworker stdout | 2023-07-06 09:01:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:01:21 UTC)" (scheduled at 2023-07-06 09:01:06.979497+00:00) quotatotalworker stdout | 2023-07-06 09:01:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 09:01:06,990 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 09:01:06,990 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:01:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 09:01:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 09:01:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 09:01:36.979497+00:00 (in 14.999606 seconds) quotatotalworker stdout | 2023-07-06 09:01:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:01:36 UTC)" (scheduled at 2023-07-06 09:01:21.979497+00:00) quotatotalworker stdout | 2023-07-06 09:01:21,980 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 09:01:21,987 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 09:01:21,987 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:01:36 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 09:01:22,469 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 09:01:22,850 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 09:01:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 09:01:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 09:02:34.887001+00:00 (in 59.999488 seconds) quotaregistrysizeworker stdout | 2023-07-06 09:01:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 09:02:34 UTC)" (scheduled at 2023-07-06 09:01:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 09:01:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 09:01:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 09:01:34,897 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 09:02:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 09:01:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 09:01:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 09:01:51.979497+00:00 (in 14.999563 seconds) quotatotalworker stdout | 2023-07-06 09:01:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:01:51 UTC)" (scheduled at 2023-07-06 09:01:36.979497+00:00) quotatotalworker stdout | 2023-07-06 09:01:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 09:01:37,000 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 09:01:37,000 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:01:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 09:01:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 09:01:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 09:02:06.979497+00:00 (in 14.999584 seconds) quotatotalworker stdout | 2023-07-06 09:01:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:02:06 UTC)" (scheduled at 2023-07-06 09:01:51.979497+00:00) quotatotalworker stdout | 2023-07-06 09:01:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 09:01:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 09:01:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:02:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 09:01:52,499 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 09:01:52,882 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 09:02:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 09:02:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 09:02:21.979497+00:00 (in 14.999577 seconds) quotatotalworker stdout | 2023-07-06 09:02:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:02:21 UTC)" (scheduled at 2023-07-06 09:02:06.979497+00:00) quotatotalworker stdout | 2023-07-06 09:02:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 09:02:06,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 09:02:06,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:02:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 09:02:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 09:02:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 09:02:36.979497+00:00 (in 14.999586 seconds) quotatotalworker stdout | 2023-07-06 09:02:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:02:36 UTC)" (scheduled at 2023-07-06 09:02:21.979497+00:00) quotatotalworker stdout | 2023-07-06 09:02:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 09:02:21,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 09:02:21,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:02:36 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 09:02:22,510 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 09:02:22,909 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 09:02:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 09:02:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 09:03:34.887001+00:00 (in 59.999485 seconds) quotaregistrysizeworker stdout | 2023-07-06 09:02:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 09:03:34 UTC)" (scheduled at 2023-07-06 09:02:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 09:02:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 09:02:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 09:02:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 09:03:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 09:02:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 09:02:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 09:02:51.979497+00:00 (in 14.999562 seconds) quotatotalworker stdout | 2023-07-06 09:02:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:02:51 UTC)" (scheduled at 2023-07-06 09:02:36.979497+00:00) quotatotalworker stdout | 2023-07-06 09:02:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 09:02:36,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 09:02:36,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:02:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 09:02:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 09:02:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 09:03:06.979497+00:00 (in 14.999586 seconds) quotatotalworker stdout | 2023-07-06 09:02:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:03:06 UTC)" (scheduled at 2023-07-06 09:02:51.979497+00:00) quotatotalworker stdout | 2023-07-06 09:02:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 09:02:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 09:02:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:03:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 09:02:52,530 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 09:02:52,942 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 09:03:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 09:03:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 09:03:21.979497+00:00 (in 14.999604 seconds) quotatotalworker stdout | 2023-07-06 09:03:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:03:21 UTC)" (scheduled at 2023-07-06 09:03:06.979497+00:00) quotatotalworker stdout | 2023-07-06 09:03:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 09:03:06,990 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 09:03:06,990 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:03:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 09:03:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 09:03:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 09:03:36.979497+00:00 (in 14.999587 seconds) quotatotalworker stdout | 2023-07-06 09:03:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:03:36 UTC)" (scheduled at 2023-07-06 09:03:21.979497+00:00) quotatotalworker stdout | 2023-07-06 09:03:21,980 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 09:03:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 09:03:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:03:36 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 09:03:22,562 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 09:03:22,974 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 09:03:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 09:03:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 09:04:34.887001+00:00 (in 59.999522 seconds) quotaregistrysizeworker stdout | 2023-07-06 09:03:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 09:04:34 UTC)" (scheduled at 2023-07-06 09:03:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 09:03:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 09:03:34,908 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 09:03:34,908 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 09:04:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 09:03:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 09:03:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 09:03:51.979497+00:00 (in 14.999571 seconds) quotatotalworker stdout | 2023-07-06 09:03:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:03:51 UTC)" (scheduled at 2023-07-06 09:03:36.979497+00:00) quotatotalworker stdout | 2023-07-06 09:03:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 09:03:36,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 09:03:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:03:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 09:03:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 09:03:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 09:04:06.979497+00:00 (in 14.999596 seconds) quotatotalworker stdout | 2023-07-06 09:03:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:04:06 UTC)" (scheduled at 2023-07-06 09:03:51.979497+00:00) quotatotalworker stdout | 2023-07-06 09:03:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 09:03:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 09:03:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:04:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 09:03:52,592 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 09:03:52,993 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 09:04:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 09:04:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 09:04:21.979497+00:00 (in 14.999542 seconds) quotatotalworker stdout | 2023-07-06 09:04:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:04:21 UTC)" (scheduled at 2023-07-06 09:04:06.979497+00:00) quotatotalworker stdout | 2023-07-06 09:04:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 09:04:06,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 09:04:06,990 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:04:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 09:04:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 09:04:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 09:04:36.979497+00:00 (in 14.999580 seconds) quotatotalworker stdout | 2023-07-06 09:04:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:04:36 UTC)" (scheduled at 2023-07-06 09:04:21.979497+00:00) quotatotalworker stdout | 2023-07-06 09:04:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 09:04:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 09:04:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:04:36 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 09:04:22,613 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 09:04:23,025 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 09:04:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 09:04:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 09:05:34.887001+00:00 (in 59.999502 seconds) quotaregistrysizeworker stdout | 2023-07-06 09:04:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 09:05:34 UTC)" (scheduled at 2023-07-06 09:04:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 09:04:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 09:04:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 09:04:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 09:05:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 09:04:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 09:04:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 09:04:51.979497+00:00 (in 14.999579 seconds) quotatotalworker stdout | 2023-07-06 09:04:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:04:51 UTC)" (scheduled at 2023-07-06 09:04:36.979497+00:00) quotatotalworker stdout | 2023-07-06 09:04:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 09:04:36,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 09:04:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:04:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 09:04:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 09:04:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 09:05:06.979497+00:00 (in 14.999578 seconds) quotatotalworker stdout | 2023-07-06 09:04:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:05:06 UTC)" (scheduled at 2023-07-06 09:04:51.979497+00:00) quotatotalworker stdout | 2023-07-06 09:04:51,980 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 09:04:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 09:04:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:05:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 09:04:52,636 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 09:04:53,053 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 09:05:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 09:05:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 09:05:21.979497+00:00 (in 14.999576 seconds) quotatotalworker stdout | 2023-07-06 09:05:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:05:21 UTC)" (scheduled at 2023-07-06 09:05:06.979497+00:00) quotatotalworker stdout | 2023-07-06 09:05:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 09:05:06,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 09:05:06,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:05:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 09:05:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 09:05:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 09:05:36.979497+00:00 (in 14.999571 seconds) quotatotalworker stdout | 2023-07-06 09:05:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:05:36 UTC)" (scheduled at 2023-07-06 09:05:21.979497+00:00) quotatotalworker stdout | 2023-07-06 09:05:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 09:05:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 09:05:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:05:36 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 09:05:22,668 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 09:05:23,075 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 09:05:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 09:05:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 09:06:34.887001+00:00 (in 59.999489 seconds) quotaregistrysizeworker stdout | 2023-07-06 09:05:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 09:06:34 UTC)" (scheduled at 2023-07-06 09:05:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 09:05:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 09:05:34,895 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 09:05:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 09:06:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 09:05:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 09:05:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 09:05:51.979497+00:00 (in 14.999567 seconds) quotatotalworker stdout | 2023-07-06 09:05:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:05:51 UTC)" (scheduled at 2023-07-06 09:05:36.979497+00:00) quotatotalworker stdout | 2023-07-06 09:05:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 09:05:36,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 09:05:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:05:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 09:05:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 09:05:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 09:06:06.979497+00:00 (in 14.999594 seconds) quotatotalworker stdout | 2023-07-06 09:05:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:06:06 UTC)" (scheduled at 2023-07-06 09:05:51.979497+00:00) quotatotalworker stdout | 2023-07-06 09:05:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 09:05:51,996 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 09:05:51,996 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:06:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 09:05:52,700 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 09:05:53,107 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 09:06:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 09:06:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 09:06:21.979497+00:00 (in 14.999548 seconds) quotatotalworker stdout | 2023-07-06 09:06:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:06:21 UTC)" (scheduled at 2023-07-06 09:06:06.979497+00:00) quotatotalworker stdout | 2023-07-06 09:06:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 09:06:06,990 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 09:06:06,990 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:06:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 09:06:21,980 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 09:06:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 09:06:36.979497+00:00 (in 14.999098 seconds) quotatotalworker stdout | 2023-07-06 09:06:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:06:36 UTC)" (scheduled at 2023-07-06 09:06:21.979497+00:00) quotatotalworker stdout | 2023-07-06 09:06:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 09:06:21,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 09:06:21,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:06:36 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 09:06:22,722 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 09:06:23,137 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 09:06:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 09:06:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 09:07:34.887001+00:00 (in 59.999533 seconds) quotaregistrysizeworker stdout | 2023-07-06 09:06:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 09:07:34 UTC)" (scheduled at 2023-07-06 09:06:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 09:06:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 09:06:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 09:06:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 09:07:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 09:06:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 09:06:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 09:06:51.979497+00:00 (in 14.999579 seconds) quotatotalworker stdout | 2023-07-06 09:06:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:06:51 UTC)" (scheduled at 2023-07-06 09:06:36.979497+00:00) quotatotalworker stdout | 2023-07-06 09:06:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 09:06:36,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 09:06:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:06:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 09:06:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 09:06:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 09:07:06.979497+00:00 (in 14.999555 seconds) quotatotalworker stdout | 2023-07-06 09:06:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:07:06 UTC)" (scheduled at 2023-07-06 09:06:51.979497+00:00) quotatotalworker stdout | 2023-07-06 09:06:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 09:06:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 09:06:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:07:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 09:06:52,754 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 09:06:53,140 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 09:07:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 09:07:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 09:07:21.979497+00:00 (in 14.999530 seconds) quotatotalworker stdout | 2023-07-06 09:07:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:07:21 UTC)" (scheduled at 2023-07-06 09:07:06.979497+00:00) quotatotalworker stdout | 2023-07-06 09:07:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 09:07:06,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 09:07:06,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:07:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 09:07:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 09:07:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 09:07:36.979497+00:00 (in 14.999552 seconds) quotatotalworker stdout | 2023-07-06 09:07:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:07:36 UTC)" (scheduled at 2023-07-06 09:07:21.979497+00:00) quotatotalworker stdout | 2023-07-06 09:07:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 09:07:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 09:07:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:07:36 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 09:07:22,771 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 09:07:23,173 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 09:07:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 09:07:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 09:08:34.887001+00:00 (in 59.999571 seconds) quotaregistrysizeworker stdout | 2023-07-06 09:07:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 09:08:34 UTC)" (scheduled at 2023-07-06 09:07:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 09:07:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 09:07:34,895 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 09:07:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 09:08:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 09:07:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 09:07:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 09:07:51.979497+00:00 (in 14.999527 seconds) quotatotalworker stdout | 2023-07-06 09:07:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:07:51 UTC)" (scheduled at 2023-07-06 09:07:36.979497+00:00) quotatotalworker stdout | 2023-07-06 09:07:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 09:07:36,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 09:07:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:07:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 09:07:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 09:07:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 09:08:06.979497+00:00 (in 14.999555 seconds) quotatotalworker stdout | 2023-07-06 09:07:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:08:06 UTC)" (scheduled at 2023-07-06 09:07:51.979497+00:00) quotatotalworker stdout | 2023-07-06 09:07:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 09:07:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 09:07:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:08:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 09:07:52,795 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 09:07:53,205 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 09:08:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 09:08:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 09:08:21.979497+00:00 (in 14.999585 seconds) quotatotalworker stdout | 2023-07-06 09:08:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:08:21 UTC)" (scheduled at 2023-07-06 09:08:06.979497+00:00) quotatotalworker stdout | 2023-07-06 09:08:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 09:08:06,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 09:08:06,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:08:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 09:08:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 09:08:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 09:08:36.979497+00:00 (in 14.999568 seconds) quotatotalworker stdout | 2023-07-06 09:08:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:08:36 UTC)" (scheduled at 2023-07-06 09:08:21.979497+00:00) quotatotalworker stdout | 2023-07-06 09:08:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 09:08:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 09:08:21,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:08:36 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 09:08:22,816 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 09:08:23,221 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 09:08:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 09:08:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 09:09:34.887001+00:00 (in 59.999520 seconds) quotaregistrysizeworker stdout | 2023-07-06 09:08:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 09:09:34 UTC)" (scheduled at 2023-07-06 09:08:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 09:08:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 09:08:34,905 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 09:08:34,905 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 09:09:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 09:08:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 09:08:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 09:08:51.979497+00:00 (in 14.999562 seconds) quotatotalworker stdout | 2023-07-06 09:08:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:08:51 UTC)" (scheduled at 2023-07-06 09:08:36.979497+00:00) quotatotalworker stdout | 2023-07-06 09:08:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 09:08:36,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 09:08:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:08:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 09:08:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 09:08:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 09:09:06.979497+00:00 (in 14.999591 seconds) quotatotalworker stdout | 2023-07-06 09:08:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:09:06 UTC)" (scheduled at 2023-07-06 09:08:51.979497+00:00) quotatotalworker stdout | 2023-07-06 09:08:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 09:08:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 09:08:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:09:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 09:08:52,848 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 09:08:53,226 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 09:09:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 09:09:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 09:09:21.979497+00:00 (in 14.999537 seconds) quotatotalworker stdout | 2023-07-06 09:09:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:09:21 UTC)" (scheduled at 2023-07-06 09:09:06.979497+00:00) quotatotalworker stdout | 2023-07-06 09:09:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 09:09:06,990 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 09:09:06,990 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:09:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 09:09:21,980 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 09:09:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 09:09:36.979497+00:00 (in 14.999302 seconds) quotatotalworker stdout | 2023-07-06 09:09:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:09:36 UTC)" (scheduled at 2023-07-06 09:09:21.979497+00:00) quotatotalworker stdout | 2023-07-06 09:09:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 09:09:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 09:09:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:09:36 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 09:09:22,860 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 09:09:23,239 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 09:09:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 09:09:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 09:10:34.887001+00:00 (in 59.999494 seconds) quotaregistrysizeworker stdout | 2023-07-06 09:09:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 09:10:34 UTC)" (scheduled at 2023-07-06 09:09:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 09:09:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 09:09:34,896 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 09:09:34,896 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 09:10:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 09:09:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 09:09:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 09:09:51.979497+00:00 (in 14.999546 seconds) quotatotalworker stdout | 2023-07-06 09:09:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:09:51 UTC)" (scheduled at 2023-07-06 09:09:36.979497+00:00) quotatotalworker stdout | 2023-07-06 09:09:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 09:09:36,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 09:09:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:09:51 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 09:09:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 09:09:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 09:10:06.979497+00:00 (in 14.999582 seconds) quotatotalworker stdout | 2023-07-06 09:09:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:10:06 UTC)" (scheduled at 2023-07-06 09:09:51.979497+00:00) quotatotalworker stdout | 2023-07-06 09:09:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 09:09:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 09:09:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:10:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 09:09:52,892 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 09:09:53,270 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 09:10:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 09:10:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 09:10:21.979497+00:00 (in 14.999563 seconds) quotatotalworker stdout | 2023-07-06 09:10:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:10:21 UTC)" (scheduled at 2023-07-06 09:10:06.979497+00:00) quotatotalworker stdout | 2023-07-06 09:10:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 09:10:06,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 09:10:06,990 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:10:21 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 09:10:21,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 09:10:21,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 09:10:36.979497+00:00 (in 14.999573 seconds) quotatotalworker stdout | 2023-07-06 09:10:21,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:10:36 UTC)" (scheduled at 2023-07-06 09:10:21.979497+00:00) quotatotalworker stdout | 2023-07-06 09:10:21,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 09:10:21,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 09:10:21,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:10:36 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 09:10:22,923 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 09:10:23,302 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 09:10:34,887 [32] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2023-07-06 09:10:34,887 [32] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 09:11:34.887001+00:00 (in 59.999528 seconds) quotaregistrysizeworker stdout | 2023-07-06 09:10:34,887 [32] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 09:11:34 UTC)" (scheduled at 2023-07-06 09:10:34.887001+00:00) quotaregistrysizeworker stdout | 2023-07-06 09:10:34,888 [32] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2023-07-06 09:10:34,895 [32] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2023-07-06 09:10:34,895 [32] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2023-07-06 09:11:34 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 09:10:36,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 09:10:36,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 09:10:51.979497+00:00 (in 14.999526 seconds) quotatotalworker stdout | 2023-07-06 09:10:36,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:10:51 UTC)" (scheduled at 2023-07-06 09:10:36.979497+00:00) quotatotalworker stdout | 2023-07-06 09:10:36,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 09:10:36,989 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 09:10:36,989 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:10:51 UTC)" executed successfully config-editor stdout | time="2023-07-06T09:10:41Z" level=debug msg="Received config bundle. Decoding into ConfigBundle struct" config-editor stdout | time="2023-07-06T09:10:42Z" level=debug msg="Validating RepoMirror" config-editor stdout | time="2023-07-06T09:10:42Z" level=debug msg="Validating QuayDocumentation" config-editor stdout | time="2023-07-06T09:10:42Z" level=debug msg="Validating LDAP" config-editor stdout | time="2023-07-06T09:10:42Z" level=debug msg="Validating JWTAuthentication" config-editor stdout | time="2023-07-06T09:10:42Z" level=debug msg="Validating Redis" config-editor stdout | time="2023-07-06T09:10:42Z" level=debug msg="Address: quayregistry-quay-redis:6379" config-editor stdout | time="2023-07-06T09:10:42Z" level=debug msg="Username: " config-editor stdout | time="2023-07-06T09:10:42Z" level=debug msg="Password Len: 0" config-editor stdout | time="2023-07-06T09:10:42Z" level=debug msg="Ssl: " config-editor stdout | time="2023-07-06T09:10:42Z" level=debug msg="Address: quayregistry-quay-redis:6379" config-editor stdout | time="2023-07-06T09:10:42Z" level=debug msg="Username: " config-editor stdout | time="2023-07-06T09:10:42Z" level=debug msg="Password Len: 0" config-editor stdout | time="2023-07-06T09:10:42Z" level=debug msg="Ssl: " config-editor stdout | time="2023-07-06T09:10:42Z" level=debug msg="Validating OIDC" config-editor stdout | time="2023-07-06T09:10:42Z" level=debug msg="Validating SecurityScanner" config-editor stdout | time="2023-07-06T09:10:42Z" level=debug msg="Validating UserVisibleSettings" config-editor stdout | time="2023-07-06T09:10:42Z" level=debug msg="Validating HostSettings" config-editor stdout | time="2023-07-06T09:10:42Z" level=debug msg="Validating TimeMachine" config-editor stdout | time="2023-07-06T09:10:42Z" level=debug msg="Validating Email" config-editor stdout | time="2023-07-06T09:10:42Z" level=debug msg="Validating BitbucketBuildTrigger" config-editor stdout | time="2023-07-06T09:10:42Z" level=debug msg="Validating ActionLogArchiving" config-editor stdout | time="2023-07-06T09:10:42Z" level=debug msg="Validating GoogleLogin" config-editor stdout | time="2023-07-06T09:10:42Z" level=debug msg="Validating AppTokenAuthentication" config-editor stdout | time="2023-07-06T09:10:42Z" level=debug msg="Validating Database" config-editor stdout | time="2023-07-06T09:10:42Z" level=debug msg="Scheme: postgresql" config-editor stdout | time="2023-07-06T09:10:42Z" level=debug msg="Host: quayregistry-quay-database:5432" config-editor stdout | time="2023-07-06T09:10:42Z" level=debug msg="Db: quayregistry-quay-database" config-editor stdout | time="2023-07-06T09:10:42Z" level=debug msg="Params: " config-editor stdout | time="2023-07-06T09:10:42Z" level=debug msg="Including params " config-editor stdout | time="2023-07-06T09:10:42Z" level=debug msg="Pinging database at postgresql://quayregistry-quay-database:C4W-Pw0aAXWhEJvzIwmZyKj08K4E5UwNDffiTF6CHMh3hVP0mu7GTCjTGm83X8a902Ypz1lKmnoTO9nM@quayregistry-quay-database:5432/quayregistry-quay-database" config-editor stdout | plpgsql config-editor stdout | pg_trgm config-editor stdout | time="2023-07-06T09:10:42Z" level=debug msg="Validating TeamSyncing" config-editor stdout | time="2023-07-06T09:10:42Z" level=debug msg="Validating DistributedStorage" config-editor stdout | time="2023-07-06T09:10:42Z" level=debug msg="Validating AccessSettings" config-editor stdout | time="2023-07-06T09:10:42Z" level=debug msg="Validating ElasticSearch" config-editor stdout | e Get "https://ec2-3-219-234-246.compute-1.amazonaws.com:9200/logentry_*": http: server gave HTTP response to HTTPS client config-editor stdout | 2023/07/06 09:10:42 [quayregistry-quay-config-editor-54d6b9bbdd-7jscr/0IZ18z1A1r-000001] "POST http://quayregistry-quay-config-editor-quay-enterprise39.apps.whu413az19.qe.azure.devcluster.openshift.com/api/v1/config/validate?mode=online HTTP/1.1" from 209.132.188.14 - 000 0B in 818.380279ms config-editor stderr | 2023/07/06 09:10:42 http: panic serving 10.129.2.24:38652: runtime error: slice bounds out of range [-1:] config-editor stderr | goroutine 50 [running]: config-editor stderr | net/http.(*conn).serve.func1() config-editor stderr | /usr/lib/golang/src/net/http/server.go:1802 +0xb9 config-editor stderr | panic({0xe5f4c0, 0xc000e8e168}) config-editor stderr | /usr/lib/golang/src/runtime/panic.go:1047 +0x266 config-editor stderr | github.com/go-chi/chi/middleware.prettyStack.decorateFuncCallLine({}, {0xc0010882d2, 0x4531d4}, 0x49, 0x8) config-editor stderr | /remote-source/config-tool/deps/gomod/pkg/mod/github.com/go-chi/chi@v4.1.2+incompatible/middleware/recoverer.go:130 +0x545 config-editor stderr | github.com/go-chi/chi/middleware.prettyStack.decorateLine({}, {0xc0010882d2, 0xc000f10600}, 0x80, 0x1) config-editor stderr | /remote-source/config-tool/deps/gomod/pkg/mod/github.com/go-chi/chi@v4.1.2+incompatible/middleware/recoverer.go:106 +0x15d config-editor stderr | github.com/go-chi/chi/middleware.prettyStack.parse({}, {0xc001080000, 0x1171, 0xc000eaad48}, {0xdc9ae0, 0x1f1ba00}) config-editor stderr | /remote-source/config-tool/deps/gomod/pkg/mod/github.com/go-chi/chi@v4.1.2+incompatible/middleware/recoverer.go:89 +0x510 config-editor stderr | github.com/go-chi/chi/middleware.PrintPrettyStack({0xdc9ae0, 0x1f1ba00}) config-editor stderr | /remote-source/config-tool/deps/gomod/pkg/mod/github.com/go-chi/chi@v4.1.2+incompatible/middleware/recoverer.go:46 +0x45 config-editor stderr | github.com/go-chi/chi/middleware.(*defaultLogEntry).Panic(0x300000002, {0xdc9ae0, 0x1f1ba00}, {0x46985b, 0xc0003611e0, 0x200000003}) config-editor stderr | /remote-source/config-tool/deps/gomod/pkg/mod/github.com/go-chi/chi@v4.1.2+incompatible/middleware/logger.go:154 +0x29 config-editor stderr | github.com/go-chi/chi/middleware.Recoverer.func1.1() config-editor stderr | /remote-source/config-tool/deps/gomod/pkg/mod/github.com/go-chi/chi@v4.1.2+incompatible/middleware/recoverer.go:28 +0xcc config-editor stderr | panic({0xdc9ae0, 0x1f1ba00}) config-editor stderr | /usr/lib/golang/src/runtime/panic.go:1038 +0x215 config-editor stderr | github.com/quay/config-tool/pkg/lib/shared.ValidateElasticSearchCredentials({0xc00065c000, 0xc000eab380}, {0xc00003c658, 0x7}, {0xc0000386a8, 0x14}) config-editor stderr | /remote-source/config-tool/app/pkg/lib/shared/validators.go:654 +0x24b config-editor stderr | github.com/quay/config-tool/pkg/lib/fieldgroups/elasticsearch.(*ElasticSearchFieldGroup).Validate(0xc00012f458, {{0xc000168262, 0xecdad4}, 0xc0006908a0}) config-editor stderr | /remote-source/config-tool/app/pkg/lib/fieldgroups/elasticsearch/elasticsearch_validator.go:93 +0x973 config-editor stderr | github.com/quay/config-tool/pkg/lib/config.Config.Validate(0xe500a0, {{0xc000168262, 0xec3c41}, 0xc0006908a0}) config-editor stderr | /remote-source/config-tool/app/pkg/lib/config/config_validator.go:20 +0x182 config-editor stderr | github.com/quay/config-tool/pkg/lib/editor.validateConfigBundle.func1({0x7f579006b0b8, 0xc00009e200}, 0xc00031c000) config-editor stderr | /remote-source/config-tool/app/pkg/lib/editor/controllers.go:213 +0x211 config-editor stderr | github.com/abbot/go-http-auth.JustCheck.func1({0x7f579006b0b8, 0xc00009e200}, 0xc00031c000) config-editor stderr | /remote-source/config-tool/deps/gomod/pkg/mod/github.com/abbot/go-http-auth@v0.4.0/auth.go:107 +0x10e config-editor stderr | github.com/abbot/go-http-auth.(*BasicAuth).Wrap.func1({0x7f579006b0b8, 0xc00009e200}, 0xc00013cc00) config-editor stderr | /remote-source/config-tool/deps/gomod/pkg/mod/github.com/abbot/go-http-auth@v0.4.0/basic.go:146 +0x11c config-editor stderr | net/http.HandlerFunc.ServeHTTP(0xdc0400, {0x7f579006b0b8, 0xc00009e200}, 0xc000168245) config-editor stderr | /usr/lib/golang/src/net/http/server.go:2047 +0x2f config-editor stderr | github.com/go-chi/chi.(*Mux).routeHTTP(0xc0001226c0, {0x7f579006b0b8, 0xc00009e200}, 0xc00013cc00) config-editor stderr | /remote-source/config-tool/deps/gomod/pkg/mod/github.com/go-chi/chi@v4.1.2+incompatible/mux.go:431 +0x1f9 config-editor stderr | net/http.HandlerFunc.ServeHTTP(0xc0000f57b0, {0x7f579006b0b8, 0xc00009e200}, 0xc2828f) config-editor stderr | /usr/lib/golang/src/net/http/server.go:2047 +0x2f config-editor stderr | github.com/go-chi/chi/middleware.Recoverer.func1({0x7f579006b0b8, 0xc00009e200}, 0xc00009e200) config-editor stderr | /remote-source/config-tool/deps/gomod/pkg/mod/github.com/go-chi/chi@v4.1.2+incompatible/middleware/recoverer.go:37 +0x83 config-editor stderr | net/http.HandlerFunc.ServeHTTP(0xc00013cb00, {0x7f579006b0b8, 0xc00009e200}, 0xa) config-editor stderr | /usr/lib/golang/src/net/http/server.go:2047 +0x2f config-editor stderr | github.com/go-chi/chi/middleware.RequestLogger.func1.1({0x109e400, 0xc0000120e0}, 0xc00013cb00) config-editor stderr | /remote-source/config-tool/deps/gomod/pkg/mod/github.com/go-chi/chi@v4.1.2+incompatible/middleware/logger.go:46 +0x453 config-editor stderr | net/http.HandlerFunc.ServeHTTP(0xc00013cb00, {0x109e400, 0xc0000120e0}, 0xc00028f440) config-editor stderr | /usr/lib/golang/src/net/http/server.go:2047 +0x2f config-editor stderr | github.com/go-chi/chi/middleware.RealIP.func1({0x109e400, 0xc0000120e0}, 0xc00013cb00) config-editor stderr | /remote-source/config-tool/deps/gomod/pkg/mod/github.com/go-chi/chi@v4.1.2+incompatible/middleware/realip.go:34 +0x9e config-editor stderr | net/http.HandlerFunc.ServeHTTP(0x10a5a78, {0x109e400, 0xc0000120e0}, 0x107b778) config-editor stderr | /usr/lib/golang/src/net/http/server.go:2047 +0x2f config-editor stderr | github.com/go-chi/chi/middleware.RequestID.func1({0x109e400, 0xc0000120e0}, 0xc00013ca00) config-editor stderr | /remote-source/config-tool/deps/gomod/pkg/mod/github.com/go-chi/chi@v4.1.2+incompatible/middleware/request_id.go:76 +0x356 config-editor stderr | net/http.HandlerFunc.ServeHTTP(0x10a59d0, {0x109e400, 0xc0000120e0}, 0x1f1b660) config-editor stderr | /usr/lib/golang/src/net/http/server.go:2047 +0x2f config-editor stderr | github.com/go-chi/chi.(*Mux).ServeHTTP(0xc0001226c0, {0x109e400, 0xc0000120e0}, 0xc00013c100) config-editor stderr | /remote-source/config-tool/deps/gomod/pkg/mod/github.com/go-chi/chi@v4.1.2+incompatible/mux.go:86 +0x3c2 config-editor stderr | net/http.serverHandler.ServeHTTP({0x109a9b0}, {0x109e400, 0xc0000120e0}, 0xc00013c100) config-editor stderr | /usr/lib/golang/src/net/http/server.go:2879 +0x43b config-editor stderr | net/http.(*conn).serve(0xc0000320a0, {0x10a5a78, 0xc0006902a0}) config-editor stderr | /usr/lib/golang/src/net/http/server.go:1930 +0xb08 config-editor stderr | created by net/http.(*Server).Serve config-editor stderr | /usr/lib/golang/src/net/http/server.go:3034 +0x4e8 quotatotalworker stdout | 2023-07-06 09:10:51,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 09:10:51,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 09:11:06.979497+00:00 (in 14.999496 seconds) quotatotalworker stdout | 2023-07-06 09:10:51,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:11:06 UTC)" (scheduled at 2023-07-06 09:10:51.979497+00:00) quotatotalworker stdout | 2023-07-06 09:10:51,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 09:10:51,988 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 09:10:51,988 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:11:06 UTC)" executed successfully quotatotalworker stdout | 2023-07-06 09:10:52,929 [33] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotatotalworker.py', 'pid': '33'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotatotalworker stdout | raise err quotatotalworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2023-07-06 09:10:53,329 [32] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-54d6b9bbdd-7jscr', 'process_name': 'quotaregistrysizeworker.py', 'pid': '32'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1285, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 980, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/http/client.py", line 946, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 844, in create_connection quotaregistrysizeworker stdout | raise err quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/socket.py", line 832, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 142, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 517, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: quotatotalworker stdout | 2023-07-06 09:11:06,979 [33] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2023-07-06 09:11:06,980 [33] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-07-06 09:11:21.979497+00:00 (in 14.999536 seconds) quotatotalworker stdout | 2023-07-06 09:11:06,980 [33] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:11:21 UTC)" (scheduled at 2023-07-06 09:11:06.979497+00:00) quotatotalworker stdout | 2023-07-06 09:11:06,981 [33] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2023-07-06 09:11:07,001 [33] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2023-07-06 09:11:07,001 [33] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2023-07-06 09:11:21 UTC)" executed successfully