Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-1709

Upgrading from an older operator with edge route breaks Quay

    XMLWordPrintable

Details

    Description

      The new operator creates self signes certificates even though Quay's config.yaml file has EXTERNAL_TLS_TERMINATION: true set (since the route is of an edge type) and puts them in the Quay's config bundle. This in turn creates an unstable configuration where nginx is bound to port 8443 and not 8080 what is expected and that causes the internal health check to break:

      gunicorn-web stdout | urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='localhost', port=443): Max retries exceeded with url: /_internal_ping (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fdf84cec490>: Failed to establish a new connection: [Errno 111] Connection refused'))
      gunicorn-web stdout | During handling of the above exception, another exception occurred:
      gunicorn-web stdout | Traceback (most recent call last):
      gunicorn-web stdout |   File "/quay-registry/health/services.py", line 43, in fn
      gunicorn-web stdout |     status_code = client.get(registry_url, verify=False, timeout=2).status_code
      gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 546, in get
      gunicorn-web stdout |     return self.request('GET', url, **kwargs)
      gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 533, in request
      gunicorn-web stdout |     resp = self.send(prep, **send_kwargs)
      gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 668, in send
      gunicorn-web stdout |     history = [resp for resp in gen] if allow_redirects else []
      gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 668, in <listcomp>
      gunicorn-web stdout |     history = [resp for resp in gen] if allow_redirects else []
      gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 239, in resolve_redirects
      gunicorn-web stdout |     resp = self.send(
      gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 646, in send
      gunicorn-web stdout |     r = adapter.send(request, **kwargs)
      gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 516, in send
      gunicorn-web stdout |     raise ConnectionError(e, request=request)
      gunicorn-web stdout | requests.exceptions.ConnectionError: HTTPSConnectionPool(host='localhost', port=443): Max retries exceeded with url: /_internal_ping (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fdf84cec490>: Failed to establish a new connection: [Errno 111] Connection refused'))
      

      The only workaround is to downscale the operator, remove the offending certificates and recreate the redge routes. If the operator persists, on next reconciliation the certs will be recreated and the health check will break again.

      Attachments

        Issue Links

          Activity

            People

              rmarasch@redhat.com Ricardo Maraschini
              rhn-support-ibazulic Ivan Bazulic
              luffy zhang luffy zhang
              Votes:
              2 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: