-
Bug
-
Resolution: Unresolved
-
Major
-
None
-
4.15.0, 4.16.0
-
Quality / Stability / Reliability
-
False
-
-
None
-
Critical
-
No
-
None
-
None
-
Rejected
-
Hypershift Sprint 249, Hypershift Sprint 250
-
2
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Description of problem:
Before configuring external OIDC, HCP env is normal. After configuring external OIDC, HCP env hits two issues: 1. HC CR's authentication not propagated to HCP authentication CR. 2. HCP console becomes degraded.
This had been observed (raised in Slack channel https://redhat-internal.slack.com/archives/C060D1W96LB/p1706888598679829?thread_ts=1706785777.255879&cid=C060D1W96LB to Seth & Standa) before OCPBUGS-28625 was solved. In a new env which verifies OCPBUGS-28625 is indeed solved, the in-question two issues still exist.
Version-Release number of selected component (if applicable):
4.16.0-0.nightly-2024-02-06-144443
How reproducible:
Always
Steps to Reproduce:
1. Install fresh HCP env. Check console: $ oc get co console NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE console 4.16.0-0.nightly-2024-02-06-144443 True False False 70m 2. Login to https://entra.microsoft.com and follow below: Create a Web App named "console", add https://<HCP_console_route>/auth/callback in "Web Redirect URIs", get the App (client) ID (to be used as $CLIENT_ID), generate a secret for it and copy the secret value (to be used as $CLIENT_SECRET_VALUE). Get the OIDC URL which looks like https://login.microsoftonline.com/<tenant_ID>/v2.0 3. Prepare below variables to be used later $ HC_NAME=hypershift-ci-263266 $ MGMT_KUBECONFIG=/path/to/management_cluster/admin/kubeconfig $ HOSTED_KUBECONFIG=/path/to/hosted_cluster/admin/kubeconfig $ AUDIENCE=<App client ID> $ ISSUER_URL=https://login.microsoftonline.com/<tenant_ID>/v2.0 $ CLIENT_ID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx $ CLIENT_SECRET_VALUE="XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" $ CLIENT_SECRET_NAME=console-secret $ export KUBECONFIG=$HOSTED_KUBECONFIG $ curl -sS "$ISSUER_URL/.well-known/openid-configuration" > microsoft-entra-id-oauthMetadata $ oc create configmap oauth-meta --from-file=oauthMetadata=microsoft-entra-id-oauthMetadata -n clusters --kubeconfig $MGMT_KUBECONFIG $ oc create secret generic console-secret -n clusters --from-literal=clientSecret=$CLIENT_SECRET_VALUE --kubeconfig $MGMT_KUBECONFIG 4. Set external OIDC configuration: $ oc patch hc $HC_NAME -n clusters --kubeconfig $MGMT_KUBECONFIG --type=merge -p=" spec: configuration: authentication: oauthMetadata: name: oauth-meta oidcProviders: - claimMappings: groups: claim: groups prefix: 'oidc-groups-test:' username: claim: email prefixPolicy: Prefix prefix: prefixString: 'oidc-user-test:' issuer: audiences: - $AUDIENCE issuerURL: $ISSUER_URL name: microsoft-entra-id oidcClients: - clientID: $CLIENT_ID clientSecret: name: $CLIENT_SECRET_NAME componentName: console componentNamespace: openshift-console type: OIDC " hostedcluster.hypershift.openshift.io/hypershift-ci-261871 patched Confirm the setting is indeed patched successfully: $ oc get hc $HC_NAME -n clusters --kubeconfig $MGMT_KUBECONFIG -o json | jq '.spec.configuration.authentication' | ~/auto/json2yaml.sh oauthMetadata: name: oauth-meta oidcProviders: - claimMappings: groups: claim: groups prefix: 'oidc-groups-test:' username: claim: email prefix: prefixString: 'oidc-user-test:' prefixPolicy: Prefix issuer: audiences: - xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx issuerURL: https://login.microsoftonline.com/<tenant_ID>/v2.0 name: microsoft-entra-id oidcClients: - clientID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx clientSecret: name: console-secret componentName: console componentNamespace: openshift-console type: OIDC Confirm pods like KAS et al are restarted: $ oc get po -n clusters-$HC_NAME --kubeconfig $MGMT_KUBECONFIG --sort-by metadata.creationTimestamp ... kube-apiserver-75c9ccb999-prmj8 5/5 Running 0 16m ... Check the kube-apiserver pod logs, the setting indeed takes effect: $ oc logs --timestamps -n clusters-$HC_NAME --kubeconfig $MGMT_KUBECONFIG kube-apiserver-75c9ccb999-prmj8 -c kube-apiserver | grep '.--oidc' ... 2024-02-07T02:20:38.258382171Z I0207 02:20:38.258378 1 flags.go:64] FLAG: --oidc-client-id="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" 2024-02-07T02:20:38.258385955Z I0207 02:20:38.258382 1 flags.go:64] FLAG: --oidc-groups-claim="groups" 2024-02-07T02:20:38.258388904Z I0207 02:20:38.258386 1 flags.go:64] FLAG: --oidc-groups-prefix="oidc-groups-test:" 2024-02-07T02:20:38.258401815Z I0207 02:20:38.258389 1 flags.go:64] FLAG: --oidc-issuer-url="https://login.microsoftonline.com/<tenant_ID>/v2.0" 2024-02-07T02:20:38.258401815Z I0207 02:20:38.258399 1 flags.go:64] FLAG: --oidc-required-claim="" 2024-02-07T02:20:38.258408848Z I0207 02:20:38.258402 1 flags.go:64] FLAG: --oidc-signing-algs="[RS256]" 2024-02-07T02:20:38.258411822Z I0207 02:20:38.258407 1 flags.go:64] FLAG: --oidc-username-claim="email" 2024-02-07T02:20:38.258414787Z I0207 02:20:38.258410 1 flags.go:64] FLAG: --oidc-username-prefix="oidc-user-test:" 5. Check the HCP .well-known/oauth-authorization-server. We see OCPBUGS-28625 is indeed solved: $ curl -k $(oc whoami --show-server)/.well-known/oauth-authorization-server {"token_endpoint":"https://login.microsoftonline.com/<tenant_ID>/oauth2/v2.0/token",... 6. Hit two issues: Issue 1: Check HCP authentication CR. Above HC authentication config does not get propagated: $ oc get authentication.config cluster -o yaml apiVersion: config.openshift.io/v1 kind: Authentication metadata: annotations: include.release.openshift.io/ibm-cloud-managed: "true" include.release.openshift.io/self-managed-high-availability: "true" include.release.openshift.io/single-node-developer: "true" release.openshift.io/create-only: "true" creationTimestamp: "2024-02-07T00:56:11Z" ... spec: oauthMetadata: name: "" serviceAccountIssuer: https://aos-hypershift-xxx.s3.us-east-2.amazonaws.com/xxxxxx type: "" Issue 2: Check HCP env, console becomes degraded: $ oc get co console NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE console 4.16.0-0.nightly-2024-02-06-144443 True False True 96m OAuthClientSyncDegraded: the server is currently unable to handle the request (get oauthclients.oauth.openshift.io console)...
Actual results:
As said in above "Description of problem".
Expected results:
No above two issues. HC CR's authentication should be propagated to HCP authentication CR; HCP console should not be degraded.
Additional info: