Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-29154

After configuring external OIDC, HCP env hits two issues: HC CR's authentication not propagated to HCP authentication CR; HCP console becomes degraded

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • None
    • 4.15.0, 4.16.0
    • HyperShift
    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • Critical
    • No
    • None
    • None
    • Rejected
    • Hypershift Sprint 249, Hypershift Sprint 250
    • 2
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Description of problem:

      Before configuring external OIDC, HCP env is normal.
      After configuring external OIDC, HCP env hits two issues:
      1. HC CR's authentication not propagated to HCP authentication CR.
      2. HCP console becomes degraded.

      This had been observed (raised in Slack channel https://redhat-internal.slack.com/archives/C060D1W96LB/p1706888598679829?thread_ts=1706785777.255879&cid=C060D1W96LB to Seth & Standa) before OCPBUGS-28625 was solved. In a new env which verifies OCPBUGS-28625 is indeed solved, the in-question two issues still exist.

      Version-Release number of selected component (if applicable):

      4.16.0-0.nightly-2024-02-06-144443

      How reproducible:

      Always

      Steps to Reproduce:

      1. Install fresh HCP env. Check console:
      $ oc get co console
      NAME                                       VERSION                              AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
      console                                    4.16.0-0.nightly-2024-02-06-144443   True        False         False      70m
      
      2. Login to https://entra.microsoft.com and follow below:
      Create a Web App named "console", add https://<HCP_console_route>/auth/callback in "Web Redirect URIs", get the App (client) ID (to be used as $CLIENT_ID), generate a secret for it and copy the secret value (to be used as $CLIENT_SECRET_VALUE).
      Get the OIDC URL which looks like https://login.microsoftonline.com/<tenant_ID>/v2.0
      
      3. Prepare below variables to be used later
      $ HC_NAME=hypershift-ci-263266
      $ MGMT_KUBECONFIG=/path/to/management_cluster/admin/kubeconfig
      $ HOSTED_KUBECONFIG=/path/to/hosted_cluster/admin/kubeconfig
      $ AUDIENCE=<App client ID>
      $ ISSUER_URL=https://login.microsoftonline.com/<tenant_ID>/v2.0
      $ CLIENT_ID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
      $ CLIENT_SECRET_VALUE="XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
      $ CLIENT_SECRET_NAME=console-secret
      $ export KUBECONFIG=$HOSTED_KUBECONFIG
      $ curl -sS "$ISSUER_URL/.well-known/openid-configuration" > microsoft-entra-id-oauthMetadata
      $ oc create configmap oauth-meta --from-file=oauthMetadata=microsoft-entra-id-oauthMetadata -n clusters --kubeconfig $MGMT_KUBECONFIG
      $ oc create secret generic console-secret -n clusters --from-literal=clientSecret=$CLIENT_SECRET_VALUE --kubeconfig $MGMT_KUBECONFIG
      
      4. Set external OIDC configuration:
      $ oc patch hc $HC_NAME -n clusters --kubeconfig $MGMT_KUBECONFIG --type=merge -p="
      spec:
        configuration:
          authentication:
            oauthMetadata:
              name: oauth-meta
            oidcProviders:
            - claimMappings:
                groups:
                  claim: groups
                  prefix: 'oidc-groups-test:'
                username:
                  claim: email
                  prefixPolicy: Prefix
                  prefix:
                    prefixString: 'oidc-user-test:'
              issuer:
                audiences:
                - $AUDIENCE
                issuerURL: $ISSUER_URL
              name: microsoft-entra-id
              oidcClients:
              - clientID: $CLIENT_ID
                clientSecret:
                  name: $CLIENT_SECRET_NAME
                componentName: console
                componentNamespace: openshift-console
            type: OIDC
      "
      hostedcluster.hypershift.openshift.io/hypershift-ci-261871 patched
      
      Confirm the setting is indeed patched successfully:
      $ oc get hc $HC_NAME -n clusters --kubeconfig $MGMT_KUBECONFIG -o json | jq '.spec.configuration.authentication' | ~/auto/json2yaml.sh
      oauthMetadata:
        name: oauth-meta
      oidcProviders:
      - claimMappings:
          groups:
            claim: groups
            prefix: 'oidc-groups-test:'
          username:
            claim: email
            prefix:
              prefixString: 'oidc-user-test:'
            prefixPolicy: Prefix
        issuer:
          audiences:
          - xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
          issuerURL: https://login.microsoftonline.com/<tenant_ID>/v2.0
        name: microsoft-entra-id
        oidcClients:
        - clientID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
          clientSecret:
            name: console-secret
          componentName: console
          componentNamespace: openshift-console
      type: OIDC
      
      Confirm pods like KAS et al are restarted:
      $ oc get po -n clusters-$HC_NAME --kubeconfig $MGMT_KUBECONFIG --sort-by metadata.creationTimestamp
      ...
      kube-apiserver-75c9ccb999-prmj8                      5/5     Running   0          16m
      
      ...
      Check the kube-apiserver pod logs, the setting indeed takes effect:
      $ oc logs --timestamps -n clusters-$HC_NAME --kubeconfig $MGMT_KUBECONFIG kube-apiserver-75c9ccb999-prmj8 -c kube-apiserver | grep '.--oidc'
      ...
      2024-02-07T02:20:38.258382171Z I0207 02:20:38.258378       1 flags.go:64] FLAG: --oidc-client-id="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
      2024-02-07T02:20:38.258385955Z I0207 02:20:38.258382       1 flags.go:64] FLAG: --oidc-groups-claim="groups"
      2024-02-07T02:20:38.258388904Z I0207 02:20:38.258386       1 flags.go:64] FLAG: --oidc-groups-prefix="oidc-groups-test:"
      2024-02-07T02:20:38.258401815Z I0207 02:20:38.258389       1 flags.go:64] FLAG: --oidc-issuer-url="https://login.microsoftonline.com/<tenant_ID>/v2.0"
      2024-02-07T02:20:38.258401815Z I0207 02:20:38.258399       1 flags.go:64] FLAG: --oidc-required-claim=""
      2024-02-07T02:20:38.258408848Z I0207 02:20:38.258402       1 flags.go:64] FLAG: --oidc-signing-algs="[RS256]"
      2024-02-07T02:20:38.258411822Z I0207 02:20:38.258407       1 flags.go:64] FLAG: --oidc-username-claim="email"
      2024-02-07T02:20:38.258414787Z I0207 02:20:38.258410       1 flags.go:64] FLAG: --oidc-username-prefix="oidc-user-test:"
      
      5. Check the HCP .well-known/oauth-authorization-server. We see OCPBUGS-28625 is indeed solved:
      $ curl -k $(oc whoami --show-server)/.well-known/oauth-authorization-server
      {"token_endpoint":"https://login.microsoftonline.com/<tenant_ID>/oauth2/v2.0/token",...
      
      6. Hit two issues:
      Issue 1:
      Check HCP authentication CR. Above HC authentication config does not get propagated:
      $ oc get authentication.config cluster -o yaml
      apiVersion: config.openshift.io/v1
      kind: Authentication
      metadata:
        annotations:
          include.release.openshift.io/ibm-cloud-managed: "true"
          include.release.openshift.io/self-managed-high-availability: "true"
          include.release.openshift.io/single-node-developer: "true"
          release.openshift.io/create-only: "true"
        creationTimestamp: "2024-02-07T00:56:11Z"
      ...
      spec:
        oauthMetadata:
          name: ""
        serviceAccountIssuer: https://aos-hypershift-xxx.s3.us-east-2.amazonaws.com/xxxxxx
        type: ""
      
      Issue 2:
      Check HCP env, console becomes degraded:
      $ oc get co console
      NAME      VERSION                              AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
      console   4.16.0-0.nightly-2024-02-06-144443   True        False         True       96m     OAuthClientSyncDegraded: the server is currently unable to handle the request (get oauthclients.oauth.openshift.io console)...
      
      

      Actual results:

      As said in above "Description of problem".

      Expected results:

      No above two issues. HC CR's authentication should be propagated to HCP authentication CR; HCP console should not be degraded.

      Additional info:

          

              sjenning Seth Jennings
              xxia-1 Xingxing Xia
              None
              None
              Xingxing Xia Xingxing Xia
              None
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Created:
                Updated: