-
Bug
-
Resolution: Done-Errata
-
Critical
-
4.15.z
-
None
-
Quality / Stability / Reliability
-
False
-
-
None
-
Critical
-
No
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Description of problem:
In 4.15 cluster, after configuring external OIDC, kube-apiserver pod crashes with: run.go:74] "command failed" err="strict decoding error: unknown field \"jwt[0].claimMappings.uid\"". In 4.16 cluster, this issue is not reproduced.
Version-Release number of selected component (if applicable):
4.15.0-0.nightly-2024-02-28-013638
How reproducible:
Unsure for now.
Steps to Reproduce:
1. Install 4.15 fresh HCP env and configure external OIDC:
$ HC_NAME=hypershift-ci-267403
MGMT_KUBECONFIG=/home/xxia/my/env/xxia-hs415-267403-4.15/kubeconfig
HOSTED_KUBECONFIG=/home/xxia/my/env/xxia-hs415-267403-4.15/hypershift-ci-267403.kubeconfig
AUDIENCE=76863fb1-xxxxxx
ISSUER_URL=https://login.microsoftonline.com/xxxxxxxx/v2.0
CLIENT_ID=76863fb1-xxxxxx
CLIENT_SECRET_VALUE="xxxxxxxx"
CLIENT_SECRET_NAME=console-secret
$ curl -sS "$ISSUER_URL/.well-known/openid-configuration" > microsoft-entra-id-oauthMetadata
$ export KUBECONFIG=$HOSTED_KUBECONFIG
$ oc create configmap tested-oauth-meta --from-file=oauthMetadata=microsoft-entra-id-oauthMetadata -n clusters --kubeconfig $MGMT_KUBECONFIG
configmap/tested-oauth-meta created
$ oc patch hc $HC_NAME -n clusters --kubeconfig $MGMT_KUBECONFIG --type=merge -p="
spec:
configuration:
authentication:
oauthMetadata:
name: tested-oauth-meta
oidcProviders:
- claimMappings:
groups:
claim: groups
prefix: 'oidc-groups-test:'
username:
claim: email
prefixPolicy: Prefix
prefix:
prefixString: 'oidc-user-test:'
issuer:
audiences:
- $AUDIENCE
issuerURL: $ISSUER_URL
name: microsoft-entra-id
oidcClients:
- clientID: $CLIENT_ID
clientSecret:
name: $CLIENT_SECRET_NAME
componentName: console
componentNamespace: openshift-console
type: OIDC
"
hostedcluster.hypershift.openshift.io/hypershift-ci-267403 patched
$ oc create secret generic console-secret -n clusters --from-literal=clientSecret=$CLIENT_SECRET_VALUE --kubeconfig $MGMT_KUBECONFIG
secret/console-secret created
$ oc get authentication.config cluster -o yaml
apiVersion: config.openshift.io/v1
kind: Authentication
metadata:
...
spec:
oauthMetadata:
name: tested-oauth-meta
oidcProviders:
- claimMappings:
groups:
claim: groups
prefix: 'oidc-groups-test:'
username:
claim: email
prefix:
prefixString: 'oidc-user-test:'
prefixPolicy: Prefix
issuer:
audiences:
- 76863fb1-xxxxxx
issuerCertificateAuthority:
name: ""
issuerURL: https://login.microsoftonline.com/xxxxxxxx/v2.0
name: microsoft-entra-id
oidcClients:
- clientID: 76863fb1-xxxxxx
clientSecret:
name: console-secret
componentName: console
componentNamespace: openshift-console
serviceAccountIssuer: https://xxxxxxxx.s3.us-east-2.amazonaws.com/hypershift-ci-267403
type: OIDC
status:
oidcClients:
- componentName: console
componentNamespace: openshift-console
conditions:
- lastTransitionTime: "2024-02-28T09:20:08Z"
message: ""
reason: OIDCConfigAvailable
status: "False"
type: Degraded
- lastTransitionTime: "2024-02-28T09:20:08Z"
message: ""
reason: OIDCConfigAvailable
status: "False"
type: Progressing
- lastTransitionTime: "2024-02-28T09:20:08Z"
message: ""
reason: OIDCConfigAvailable
status: "True"
type: Available
currentOIDCClients:
- clientID: 76863fb1-xxxxxx
issuerURL: https://login.microsoftonline.com/xxxxxxxx/v2.0
oidcProviderName: microsoft-entra-id
$ oc get po -n clusters-$HC_NAME --kubeconfig $MGMT_KUBECONFIG --sort-by metadata.creationTimestamp
NAME READY STATUS RESTARTS AGE
...
openshift-apiserver-d665bdc58-7cfdg 3/3 Running 0 154m
kube-controller-manager-577cf4566f-sgxz2 1/1 Running 0 154m
openshift-apiserver-d665bdc58-52w9m 3/3 Running 0 154m
kube-apiserver-74f569dfb5-7tnmn 4/5 CrashLoopBackOff 7 (2m47s ago) 15m
$ oc logs --timestamps -n clusters-$HC_NAME --kubeconfig $MGMT_KUBECONFIG -c kube-apiserver kube-apiserver-74f569dfb5-7tnmn > ~/my/logs/kube-apiserver-74f569dfb5-7tnmn-CrashLoopBackOff-hcp415.log
$ oc get cm auth-config -n clusters-$HC_NAME --kubeconfig $MGMT_KUBECONFIG -o jsonpath='{.data.auth\.json}'
{"kind":"AuthenticationConfiguration","apiVersion":"apiserver.config.k8s.io/v1alpha1","jwt":[{"issuer":{"url":"https://login.microsoftonline.com/xxxxxxxx/v2.0","audiences":["76863fb1-xxxxxx"],"audienceMatchPolicy":"MatchAny"},"claimMappings":{"username":{"claim":"email","prefix":"oidc-user-test:"},"groups":{"claim":"groups","prefix":"oidc-groups-test:"},"uid":{}}}]}
$ oc get cm auth-config -n clusters-$HC_NAME --kubeconfig $MGMT_KUBECONFIG -o jsonpath='{.data.auth\.json}' | jq | ~/auto/json2yaml.sh
---
kind: AuthenticationConfiguration
apiVersion: apiserver.config.k8s.io/v1alpha1
jwt:
- issuer:
url: https://login.microsoftonline.com/xxxxxxxx/v2.0
audiences:
- 76863fb1-xxxxxx
audienceMatchPolicy: MatchAny
claimMappings:
username:
claim: email
prefix: 'oidc-user-test:'
groups:
claim: groups
prefix: 'oidc-groups-test:'
uid: {}
$ vi ~/my/logs/kube-apiserver-74f569dfb5-7tnmn-CrashLoopBackOff-hcp415.log
...
2024-02-28T09:32:06.077307893Z I0228 09:32:06.077298 1 options.go:220] external host was not specified, using 172.20.0.1
2024-02-28T09:32:06.077977888Z I0228 09:32:06.077952 1 server.go:189] Version: v1.28.6+6216ea1
2024-02-28T09:32:06.077977888Z I0228 09:32:06.077971 1 server.go:191] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
2024-02-28T09:32:06.078556862Z I0228 09:32:06.078543 1 dynamic_serving_content.go:113] "Loaded a new cert/key pair" name="serving-cert::/etc/kubernetes/certs/server/tls.crt::/etc/kubernetes/certs/server/tls.key"
2024-02-28T09:32:06.408308750Z I0228 09:32:06.408274 1 dynamic_cafile_content.go:119] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/certs/client-ca/ca.crt"
2024-02-28T09:32:06.408487434Z E0228 09:32:06.408467 1 run.go:74] "command failed" err="strict decoding error: unknown field \"jwt[0].claimMappings.uid\""
Actual results:
As shown in above "Description of problem"
Expected results:
4.16 does not have the issue. 4.15 should have no such problem.
Additional info:
- links to
-
RHSA-2024:1210
OpenShift Container Platform 4.15.z security update