Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-28625

HCP .well-known/oauth-authorization-server shows "https://:0" even OIDC oauthMetadata is set in hc.spec.configuration.authentication

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done-Errata
    • Icon: Major Major
    • 4.16.0
    • 4.15.0, 4.16.0
    • HyperShift
    • Critical
    • No
    • Rejected
    • False
    • Hide

      None

      Show
      None
    • Release Note Not Required
    • In Progress

      Description of problem:

      HCP does not honor the oauthMetadata field of hc.spec.configuration.authentication, making console crash and oc login fail.
      

      Version-Release number of selected component (if applicable):

      HyperShift management cluster: 4.16.0-0.nightly-2024-01-29-233218
      HyperShift hosted cluster: 4.16.0-0.nightly-2024-01-29-233218

      How reproducible:

      Always

      Steps to Reproduce:

      1. Install HCP env. Export KUBECONFIG:
      $ export KUBECONFIG=/path/to/hosted-cluster/kubeconfig
      
      2. Create keycloak applications. Then get the route:
      $ KEYCLOAK_HOST=https://$(oc get -n keycloak route keycloak --template='{{ .spec.host }}')
      $ echo $KEYCLOAK_HOST
      https://keycloak-keycloak.apps.hypershift-ci-18556.xxx
      $ curl -sSk "$KEYCLOAK_HOST/realms/master/.well-known/openid-configuration" > oauthMetadata
      
      $ cat oauthMetadata 
      {"issuer":"https://keycloak-keycloak.apps.hypershift-ci-18556.xxx/realms/master"
      
      $ oc create configmap oauth-meta --from-file ./oauthMetadata -n clusters --kubeconfig /path/to/management-cluster/kubeconfig
      ...
      
      3. Set hc.spec.configuration.authentication:
      $ CLIENT_ID=openshift-test-aud
      $ oc patch hc hypershift-ci-18556 -n clusters --kubeconfig /path/to/management-cluster/kubeconfig --type=merge -p="
      spec:
        configuration:
          authentication:
            oauthMetadata:
              name: oauth-meta
            oidcProviders:
            - claimMappings:
                ...
              issuer:
                audiences:
                - $CLIENT_ID
                issuerCertificateAuthority:
                  name: keycloak-oidc-ca
                issuerURL: $KEYCLOAK_HOST/realms/master
              name: keycloak-oidc-test
            type: OIDC
      "
      
      Check KAS indeed already picks up the setting:
      $ oc logs -c kube-apiserver kube-apiserver-5c976d59f5-zbrwh -n clusters-hypershift-ci-18556 --kubeconfig /path/to/management-cluster/kubeconfig | grep "oidc-"
      ...
      I0130 08:07:24.266247       1 flags.go:64] FLAG: --oidc-ca-file="/etc/kubernetes/certs/oidc-ca/ca.crt"
      I0130 08:07:24.266251       1 flags.go:64] FLAG: --oidc-client-id="openshift-test-aud"
      ...
      I0130 08:07:24.266261       1 flags.go:64] FLAG: --oidc-issuer-url="https://keycloak-keycloak.apps.hypershift-ci-18556.xxx/realms/master"
      ...
      
      Wait about 15 mins.
      
      4. Check COs and check oc login. Both show the same error:
      $ oc get co | grep -v 'True.*False.*False'
      NAME                                       VERSION                              AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
      console                                    4.16.0-0.nightly-2024-01-29-233218   True        True          False      4h57m   SyncLoopRefreshProgressing: Working toward version 4.16.0-0.nightly-2024-01-29-233218, 1 replicas available
      $ oc get po -n openshift-console
      NAME                        READY   STATUS             RESTARTS         AGE
      console-547cf6bdbb-l8z9q    1/1     Running            0                4h55m
      console-54f88749d7-cv7ht    0/1     CrashLoopBackOff   9 (3m18s ago)    14m
      console-54f88749d7-t7x96    0/1     CrashLoopBackOff   9 (3m32s ago)    14m
      
      $ oc logs console-547cf6bdbb-l8z9q -n openshift-console
      I0130 03:23:36.788951       1 metrics.go:156] usage.Metrics: Update console users metrics: 0 kubeadmin, 0 cluster-admins, 0 developers, 0 unknown/errors (took 406.059196ms)
      E0130 06:48:32.745179       1 asynccache.go:43] failed a caching attempt: request to OAuth issuer endpoint https://:0/oauth/token failed: Head "https://:0": dial tcp :0: connect: connection refused
      E0130 06:53:32.757881       1 asynccache.go:43] failed a caching attempt: request to OAuth issuer endpoint https://:0/oauth/token failed: Head "https://:0": dial tcp :0: connect: connection refused
      ...
      
      $ oc login --exec-plugin=oc-oidc --client-id=openshift-test-aud --extra-scopes=email,profile --callback-port=8080
      error: oidc authenticator error: oidc discovery error: Get "https://:0/.well-known/openid-configuration": dial tcp :0: connect: connection refused
      error: oidc authenticator error: oidc discovery error: Get "https://:0/.well-known/openid-configuration": dial tcp :0: connect: connection refused
      Unable to connect to the server: getting credentials: exec: executable oc failed with exit code 1
      
      5. Check root cause, the configured oauthMetadata is not picked up well:
      $ curl -k https://a6e149f24f8xxxxxx.elb.ap-east-1.amazonaws.com:6443/.well-known/oauth-authorization-server
      {
      "issuer": "https://:0",
      "authorization_endpoint": "https://:0/oauth/authorize",
      "token_endpoint": "https://:0/oauth/token",
      ...
      }
      

      Actual results:

      As above steps 4 and 5, the configured oauthMetadata is not picked up well, causing console and oc login hit the error.

      Expected results:

      The configured oauthMetadata is picked up well. No error.

      Additional info:

      For oc, if I manually use `oc config set-credentials oidc --exec-api-version=client.authentication.k8s.io/v1 --exec-command=oc --exec-arg=get-token --exec-arg="--issuer-url=$KEYCLOAK_HOST/realms/master" ...` instead of using `oc login --exec-plugin=oc-oidc ...`, oc authentication works well. This means my configuration is correct.
      $ oc whoami  
      Please visit the following URL in your browser: http://localhost:8080
      oidc-user-test:xxia@redhat.com

              sjenning Seth Jennings
              xxia-1 Xingxing Xia
              Xingxing Xia Xingxing Xia
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated:
                Resolved: