Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-1709

Upgrading from an older operator with edge route breaks Quay

XMLWordPrintable

    • False
    • False
    • Quay Enterprise
    • Undefined

      The new operator creates self signes certificates even though Quay's config.yaml file has EXTERNAL_TLS_TERMINATION: true set (since the route is of an edge type) and puts them in the Quay's config bundle. This in turn creates an unstable configuration where nginx is bound to port 8443 and not 8080 what is expected and that causes the internal health check to break:

      gunicorn-web stdout | urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='localhost', port=443): Max retries exceeded with url: /_internal_ping (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fdf84cec490>: Failed to establish a new connection: [Errno 111] Connection refused'))
gunicorn-web stdout | During handling of the above exception, another exception occurred:
gunicorn-web stdout | Traceback (most recent call last):
gunicorn-web stdout |   File "/quay-registry/health/services.py", line 43, in fn
gunicorn-web stdout |     status_code = client.get(registry_url, verify=False, timeout=2).status_code
gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 546, in get
gunicorn-web stdout |     return self.request('GET', url, **kwargs)
gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 533, in request
gunicorn-web stdout |     resp = self.send(prep, **send_kwargs)
gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 668, in send
gunicorn-web stdout |     history = [resp for resp in gen] if allow_redirects else []
gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 668, in <listcomp>
gunicorn-web stdout |     history = [resp for resp in gen] if allow_redirects else []
gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 239, in resolve_redirects
gunicorn-web stdout |     resp = self.send(
gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 646, in send
gunicorn-web stdout |     r = adapter.send(request, **kwargs)
gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 516, in send
gunicorn-web stdout |     raise ConnectionError(e, request=request)
gunicorn-web stdout | requests.exceptions.ConnectionError: HTTPSConnectionPool(host='localhost', port=443): Max retries exceeded with url: /_internal_ping (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fdf84cec490>: Failed to establish a new connection: [Errno 111] Connection refused'))

      The only workaround is to downscale the operator, remove the offending certificates and recreate the redge routes. If the operator persists, on next reconciliation the certs will be recreated and the health check will break again.

              rmarasch@redhat.com Ricardo Maraschini (Inactive)
              rhn-support-ibazulic Ivan Bazulic
              luffy zhang luffy zhang
              Votes:
              2 Vote for this issue
              Watchers:
              11 Start watching this issue

                Created:
                Updated:
                Resolved: