Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-5925

A lot of quotatotalworker error in quayregistry-quay-config-editor pod log

XMLWordPrintable

    • False
    • None
    • False

      Description of problem:

      There are a lot of  quotatotalworker error in quayregistry-quay-config-editor pod log.  config tool should not run quotatotalworker process.

      $ oc get pod
      quay-operator.v3.9.0-59475dbcf7-pd8lr              1/1     Running     0   .....
      quayregistry-quay-config-editor-7cf74f4c9c-6gf6k   1/1     Running     0              27m
      
      
      $ oc logs quayregistry-quay-config-editor-7cf74f4c9c-6gf6k
      ..........................
      quotatotalworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-7cf74f4c9c-6gf6k', 'process_name': 'quotatotalworker.py', 'pid': '37'}
      quotatotalworker stderr | Traceback (most recent call last):
      quotatotalworker stderr |   File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open
      quotatotalworker stderr |     h.request(req.get_method(), req.selector, req.data, headers,
      quotatotalworker stderr |   File "/usr/lib64/python3.9/http/client.py", line 1285, in request
      quotatotalworker stderr |     self._send_request(method, url, body, headers, encode_chunked)
      quotatotalworker stderr |   File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request
      quotatotalworker stderr |     self.endheaders(body, encode_chunked=encode_chunked)
      quotatotalworker stderr |   File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders
      quotatotalworker stderr |     self._send_output(message_body, encode_chunked=encode_chunked)
      quotatotalworker stderr |   File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output
      quotatotalworker stderr |     self.send(msg)
      quotatotalworker stderr |   File "/usr/lib64/python3.9/http/client.py", line 980, in send
      quotatotalworker stderr |     self.connect()
      quotatotalworker stderr |   File "/usr/lib64/python3.9/http/client.py", line 946, in connect
      quotatotalworker stderr |     self.sock = self._create_connection(
      quotatotalworker stderr |   File "/usr/lib64/python3.9/socket.py", line 844, in create_connection
      quotatotalworker stderr |     raise err
      quotatotalworker stderr |   File "/usr/lib64/python3.9/socket.py", line 832, in create_connection
      quotatotalworker stderr |     sock.connect(sa)
      quotatotalworker stderr | ConnectionRefusedError: [Errno 111] Connection refused
      quotatotalworker stderr | During handling of the above exception, another exception occurred:
      quotatotalworker stderr | Traceback (most recent call last):
      quotatotalworker stderr |   File "/quay-registry/util/metrics/prometheus.py", line 142, in run
      quotatotalworker stderr |     push_to_gateway(
      quotatotalworker stderr |   File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway
      quotatotalworker stderr |     _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler)
      quotatotalworker stderr |   File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway
      quotatotalworker stderr |     handler(
      quotatotalworker stderr |   File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle
      quotatotalworker stderr |     resp = build_opener(HTTPHandler).open(request, timeout=timeout)
      quotatotalworker stderr |   File "/usr/lib64/python3.9/urllib/request.py", line 517, in open
      quotatotalworker stderr |     response = self._open(req, data)
      quotatotalworker stderr |   File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open
      quotatotalworker stderr |     result = self._call_chain(self.handle_open, protocol, protocol +
      quotatotalworker stderr |   File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain
      quotatotalworker stderr |     result = func(*args)
      quotatotalworker stderr |   File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open
      quotatotalworker stderr |     return self.do_open(http.client.HTTPConnection, req)
      quotatotalworker stderr |   File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open
      quotatotalworker stderr |     raise URLError(err)
      quotatotalworker stderr | urllib.error.URLError: <urlopen error [Errno 111] Connection refused>
      quotaregistrysizeworker stdout | 2023-08-09 10:01:09,835 [36] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-config-editor-7cf74f4c9c-6gf6k', 'process_name': 'quotaregistrysizeworker.py', 'pid': '36'}
      quotaregistrysizeworker stdout | Traceback (most recent call last):
      quotaregistrysizeworker stdout |   File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open
      quotaregistrysizeworker stdout |     h.request(req.get_method(), req.selector, req.data, headers,
      quotaregistrysizeworker stdout |   File "/usr/lib64/python3.9/http/client.py", line 1285, in request
      quotaregistrysizeworker stdout |     self._send_request(method, url, body, headers, encode_chunked)
      quotaregistrysizeworker stdout |   File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request
      quotaregistrysizeworker stdout |     self.endheaders(body, encode_chunked=encode_chunked)
      quotaregistrysizeworker stdout |   File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders
      quotaregistrysizeworker stdout |     self._send_output(message_body, encode_chunked=encode_chunked)
      quotaregistrysizeworker stdout |   File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output
      quotaregistrysizeworker stdout |     self.send(msg)
      quotaregistrysizeworker stdout |   File "/usr/lib64/python3.9/http/client.py", line 980, in send
      quotaregistrysizeworker stdout |     self.connect()
      quotaregistrysizeworker stdout |   File "/usr/lib64/python3.9/http/client.py", line 946, in connect
      quotaregistrysizeworker stdout |     self.sock = self._create_connection(
      quotaregistrysizeworker stdout |   File "/usr/lib64/python3.9/socket.py", line 844, in create_connection
      quotaregistrysizeworker stdout |     raise err
      quotaregistrysizeworker stdout |   File "/usr/lib64/python3.9/socket.py", line 832, in create_connection
      quotaregistrysizeworker stdout |     sock.connect(sa)
      quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused
      quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred:
      quotaregistrysizeworker stdout | Traceback (most recent call last):
      quotaregistrysizeworker stdout |   File "/quay-registry/util/metrics/prometheus.py", line 142, in run
      quotaregistrysizeworker stdout |     push_to_gateway(
      quotaregistrysizeworker stdout |   File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway
      quotaregistrysizeworker stdout |     _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler)
      quotaregistrysizeworker stdout |   File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway
      quotaregistrysizeworker stdout |     handler(
      quotaregistrysizeworker stdout |   File "/app/lib/python3.9/site-packages/prometheus_client/exposition.py", line 221, in handle
      quotaregistrysizeworker stdout |     resp = build_opener(HTTPHandler).open(request, timeout=timeout)
      quotaregistrysizeworker stdout |   File "/usr/lib64/python3.9/urllib/request.py", line 517, in open
      quotaregistrysizeworker stdout |     response = self._open(req, data)
      quotaregistrysizeworker stdout |   File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open
      quotaregistrysizeworker stdout |     result = self._call_chain(self.handle_open, protocol, protocol +
      quotaregistrysizeworker stdout |   File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain
      quotaregistrysizeworker stdout |     result = func(*args)
      quotaregistrysizeworker stdout |   File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open
      quotaregistrysizeworker stdout |     return self.do_open(http.client.HTTPConnection, req)
      quotaregistrysizeworker stdout |   File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open
      quotaregistrysizeworker stdout |     raise URLError(err)
      quotaregistrysizeworker stdout | urllib.error.URLError: <urlopen error [Errno 111] Connection refused>

       

              bcaton@redhat.com Brandon Caton
              rhwhu Weihua Hu
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: