Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-9868

LoadBalancer incompatibility - Quay uses SERVER_HOSTNAME as bearer realm in WWW-Authenticate header instead of request host url

XMLWordPrintable

    • Icon: Story Story
    • Resolution: Unresolved
    • Icon: Undefined Undefined
    • None
    • quay-v3.15.2
    • quay
    • False
    • Hide

      None

      Show
      None
    • False

      Our customer ueses Quay in a highly restricted and network segmented environment and exposes the registry over different Routes via dedicated LoadBalancers.

      Infrastructure Setup

      • OpenShift Cluster OCP-1 hosts quay registry Q-1
      • OpenShift Cluster OCP-1 has 3 routes, all pointing to the same quay service
        • route R-1 for internal access (operator managed)
        • route R-2 for external access (configures load balancer with dedicated hostname)
        • route R-2 for external access (configures load balancer with dedicated hostname)
      • OpenShift Cluster OCP-2 accesses Q-1 registry via route R-2 exposed via a LB that OCP-2 has access to
      • OpenShift Cluster OCP-3 accesses Q-1 registry via route R-3 exposed via a LB that OCP-3 has access to

      Problem Statement

      • When OCP-2 tries to access Q-1 via R-2 registry.external.domain.redacted they receive a 401 reponse to authenticate against the registry > OK
      • The response unfortunately contains the internal bearer realm registry.apps.ocp-1.internal.domain.redacted/v2/auth that is used in R-1 (and SERVER_HOSTNAME) > NOK
        • This realm is not reachable from the network, where OCP-2 is located in and therefore pull / auth requests fail
        • The realm should reuse the host from the initial request URL, while service parameter can/should stay as it is

      Additional Question

      • Is there any other possibility to expose the same registry via different URLs to different environments?

      Logs

      Podman

      [core@f-e-ocp01-2-svrm1 ~]$ podman pull https://registry.external.domain.redacted/repo_1/image:latest --log-level debug
      ...
      DEBU[0010] GET https://registry.external.domain.redacted/v2/
      DEBU[0010] Ping https://registry.external.domain.redacted/v2/ status 401
      DEBU[0010] GET https://registry.apps.ocp-1.internal.domain.redacted/v2/auth?account=jdoe&service=registry.apps.ocp-1.internal.domain.redacted
      Error: authenticating creds for "registry.external.domain.redacted": Get https://registry.apps.ocp-1.internal.domain.redacted/v2/auth?account=jdoe&service=registry.apps.ocp-1.internal.domain.redacted: dial tcp 10.1.1.67:443: i/o timeout
      DEBU[0040] Shutting down engines
      INFO[0040] Received shutdown.Stop(), terminating! PID=4738 

      Curl reproduceable

      [core@f-e-ocp01-2-svrm1 ~]$ curl -vI https://registry.external.domain.redacted/v2/
      
      Trying 10.2.2.28:443...
      Connected to registry.external.domain.redacted (10.2.2.28) port 443 (#0)
      ALPN, offering h2
      ALPN, offering http/1.1
      CAfile: /etc/pki/tls/certs/ca-bundle.crt
      ... TLS handshake ...
      SSL connection using TLSv1.2 / AES256-SHA
      ALPN, server accepted to use http/1.1
      Server certificate:
      subject: <redacted>
      start date: Aug 1 12:09:43 2025 GMT
      expire date: Aug 1 12:09:43 2027 GMT
      subjectAltName: host "<redacted>" matched cert's "*.<redacted>"
      issuer: <redacted>
      SSL certificate verify ok.
      TLSv1.2 (OUT), TLS header, Unknown (23):
      > HEAD /v2/ HTTP/1.1
      > Host: registry.external.domain.redacted
      > User-Agent: curl/7.76.1
      > Accept: /
      >
      TLSv1.2 (IN), TLS header, Unknown (23):
      Mark bundle as not supporting multiuse
      < HTTP/1.1 401 UNAUTHORIZED
      HTTP/1.1 401 UNAUTHORIZED
      < Server: nginx/1.22.1
      Server: nginx/1.22.1
      < Date: Wed, 26 Nov 2025 17:32:45 GMT
      Date: Wed, 26 Nov 2025 17:32:45 GMT
      < Content-Type: text/html; charset=utf-8
      Content-Type: text/html; charset=utf-8
      < Content-Length: 4
      Content-Length: 4
      < Connection: close
      Connection: close
      < WWW-Authenticate: Bearer realm=https://registry.apps.ocp-1.internal.domain.redacted/v2/auth,service="registry.apps.ocp-1.internal.domain.redacted"
      WWW-Authenticate: Bearer realm=https://registry.apps.ocp-1.internal.domain.redacted/v2/auth,service="registry.apps.ocp-1.internal.domain.redacted"
      < Docker-Distribution-API-Version: registry/2.0
      Docker-Distribution-API-Version: registry/2.0
       

              Unassigned Unassigned
              p.chlebicki@bacher.at Philipp Chlebicki
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated: