Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-30029

[4.15] After configuring external OIDC, kube-apiserver pod crashes with: run.go:74] "command failed" err="strict decoding error: unknown field \"jwt[0].claimMappings.uid\""

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done-Errata
    • Icon: Critical Critical
    • 4.15.z
    • 4.15.z
    • HyperShift
    • None

      Description of problem:

      In 4.15 cluster, after configuring external OIDC, kube-apiserver pod crashes with: run.go:74] "command failed" err="strict decoding error: unknown field \"jwt[0].claimMappings.uid\"".
      In 4.16 cluster, this issue is not reproduced.

      Version-Release number of selected component (if applicable):

      4.15.0-0.nightly-2024-02-28-013638    

      How reproducible:

      Unsure for now.

      Steps to Reproduce:

      1. Install 4.15 fresh HCP env and configure external OIDC:
      $ HC_NAME=hypershift-ci-267403
      MGMT_KUBECONFIG=/home/xxia/my/env/xxia-hs415-267403-4.15/kubeconfig
      HOSTED_KUBECONFIG=/home/xxia/my/env/xxia-hs415-267403-4.15/hypershift-ci-267403.kubeconfig
      AUDIENCE=76863fb1-xxxxxx
      ISSUER_URL=https://login.microsoftonline.com/xxxxxxxx/v2.0
      CLIENT_ID=76863fb1-xxxxxx
      CLIENT_SECRET_VALUE="xxxxxxxx"
      CLIENT_SECRET_NAME=console-secret
      
      $ curl -sS "$ISSUER_URL/.well-known/openid-configuration" > microsoft-entra-id-oauthMetadata
      
      $ export KUBECONFIG=$HOSTED_KUBECONFIG
      
      $ oc create configmap tested-oauth-meta --from-file=oauthMetadata=microsoft-entra-id-oauthMetadata -n clusters --kubeconfig $MGMT_KUBECONFIG
      configmap/tested-oauth-meta created
      
      $ oc patch hc $HC_NAME -n clusters --kubeconfig $MGMT_KUBECONFIG --type=merge -p="
      spec:
        configuration:
          authentication:
            oauthMetadata:
              name: tested-oauth-meta
            oidcProviders:
            - claimMappings:
                groups:
                  claim: groups
                  prefix: 'oidc-groups-test:'
                username:
                  claim: email
                  prefixPolicy: Prefix
                  prefix:
                    prefixString: 'oidc-user-test:'
              issuer:
                audiences:
                - $AUDIENCE
                issuerURL: $ISSUER_URL
              name: microsoft-entra-id
              oidcClients:
              - clientID: $CLIENT_ID
                clientSecret:
                  name: $CLIENT_SECRET_NAME
                componentName: console
                componentNamespace: openshift-console
            type: OIDC
      "
      hostedcluster.hypershift.openshift.io/hypershift-ci-267403 patched
      
      $ oc create secret generic console-secret -n clusters --from-literal=clientSecret=$CLIENT_SECRET_VALUE --kubeconfig $MGMT_KUBECONFIG
      secret/console-secret created
      
      $ oc get authentication.config cluster -o yaml
      apiVersion: config.openshift.io/v1
      kind: Authentication
      metadata:
      ...
      spec:
        oauthMetadata:
          name: tested-oauth-meta
        oidcProviders:
        - claimMappings:
            groups:
              claim: groups
              prefix: 'oidc-groups-test:'
            username:
              claim: email
              prefix:
                prefixString: 'oidc-user-test:'
              prefixPolicy: Prefix
          issuer:
            audiences:
            - 76863fb1-xxxxxx
            issuerCertificateAuthority:
              name: ""
            issuerURL: https://login.microsoftonline.com/xxxxxxxx/v2.0
          name: microsoft-entra-id
          oidcClients:
          - clientID: 76863fb1-xxxxxx
            clientSecret:
              name: console-secret
            componentName: console
            componentNamespace: openshift-console
        serviceAccountIssuer: https://xxxxxxxx.s3.us-east-2.amazonaws.com/hypershift-ci-267403
        type: OIDC
      status:
        oidcClients:
        - componentName: console
          componentNamespace: openshift-console
          conditions:
          - lastTransitionTime: "2024-02-28T09:20:08Z"
            message: ""
            reason: OIDCConfigAvailable
            status: "False"
            type: Degraded
          - lastTransitionTime: "2024-02-28T09:20:08Z"
            message: ""
            reason: OIDCConfigAvailable
            status: "False"
            type: Progressing
          - lastTransitionTime: "2024-02-28T09:20:08Z"
            message: ""
            reason: OIDCConfigAvailable
            status: "True"
            type: Available
          currentOIDCClients:
          - clientID: 76863fb1-xxxxxx
            issuerURL: https://login.microsoftonline.com/xxxxxxxx/v2.0
            oidcProviderName: microsoft-entra-id
      $ oc get po -n clusters-$HC_NAME --kubeconfig $MGMT_KUBECONFIG --sort-by metadata.creationTimestamp
      NAME                                                  READY   STATUS             RESTARTS        AGE
      ...
      openshift-apiserver-d665bdc58-7cfdg                   3/3     Running            0               154m
      kube-controller-manager-577cf4566f-sgxz2              1/1     Running            0               154m
      openshift-apiserver-d665bdc58-52w9m                   3/3     Running            0               154m
      kube-apiserver-74f569dfb5-7tnmn                       4/5     CrashLoopBackOff   7 (2m47s ago)   15m
      
      $ oc logs --timestamps -n clusters-$HC_NAME --kubeconfig $MGMT_KUBECONFIG -c kube-apiserver kube-apiserver-74f569dfb5-7tnmn > ~/my/logs/kube-apiserver-74f569dfb5-7tnmn-CrashLoopBackOff-hcp415.log
      
      $ oc get cm auth-config -n clusters-$HC_NAME --kubeconfig $MGMT_KUBECONFIG -o jsonpath='{.data.auth\.json}'
      {"kind":"AuthenticationConfiguration","apiVersion":"apiserver.config.k8s.io/v1alpha1","jwt":[{"issuer":{"url":"https://login.microsoftonline.com/xxxxxxxx/v2.0","audiences":["76863fb1-xxxxxx"],"audienceMatchPolicy":"MatchAny"},"claimMappings":{"username":{"claim":"email","prefix":"oidc-user-test:"},"groups":{"claim":"groups","prefix":"oidc-groups-test:"},"uid":{}}}]}
      
      $ oc get cm auth-config -n clusters-$HC_NAME --kubeconfig $MGMT_KUBECONFIG -o jsonpath='{.data.auth\.json}' | jq | ~/auto/json2yaml.sh
      ---
      kind: AuthenticationConfiguration
      apiVersion: apiserver.config.k8s.io/v1alpha1
      jwt:
      - issuer:
          url: https://login.microsoftonline.com/xxxxxxxx/v2.0
          audiences:
          - 76863fb1-xxxxxx
          audienceMatchPolicy: MatchAny
        claimMappings:
          username:
            claim: email
            prefix: 'oidc-user-test:'
          groups:
            claim: groups
            prefix: 'oidc-groups-test:'
          uid: {}
      $ vi ~/my/logs/kube-apiserver-74f569dfb5-7tnmn-CrashLoopBackOff-hcp415.log
      ...
      2024-02-28T09:32:06.077307893Z I0228 09:32:06.077298       1 options.go:220] external host was not specified, using 172.20.0.1
      2024-02-28T09:32:06.077977888Z I0228 09:32:06.077952       1 server.go:189] Version: v1.28.6+6216ea1
      2024-02-28T09:32:06.077977888Z I0228 09:32:06.077971       1 server.go:191] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
      2024-02-28T09:32:06.078556862Z I0228 09:32:06.078543       1 dynamic_serving_content.go:113] "Loaded a new cert/key pair" name="serving-cert::/etc/kubernetes/certs/server/tls.crt::/etc/kubernetes/certs/server/tls.key"
      2024-02-28T09:32:06.408308750Z I0228 09:32:06.408274       1 dynamic_cafile_content.go:119] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/certs/client-ca/ca.crt"
      2024-02-28T09:32:06.408487434Z E0228 09:32:06.408467       1 run.go:74] "command failed" err="strict decoding error: unknown field \"jwt[0].claimMappings.uid\""
           

      Actual results:

      As shown in above "Description of problem"

      Expected results:

      4.16 does not have the issue. 4.15 should have no such problem.

      Additional info:

       

       

              sjenning Seth Jennings
              xxia-1 Xingxing Xia
              Xingxing Xia Xingxing Xia
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated:
                Resolved: