Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-30991

When issuerCertificateAuthority is set, kube-apiserver gets CrashLoopBackOff

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Undefined Undefined
    • 4.16.0
    • 4.15.z, 4.16
    • HyperShift
    • Critical
    • No
    • Hypershift Sprint 251
    • 1
    • Approved
    • False
    • Hide

      None

      Show
      None

      Description of problem:

      When issuerCertificateAuthority is set, kube-apiserver pod is CrashLoopBackOff.
      
      Tried RCA debugging, found the cause is: the path /etc/kubernetes/certs/oidc-ca/ca.crt is incorrect. The expected path should be /etc/kubernetes/certs/oidc-ca/ca-bundle.crt .    

      Version-Release number of selected component (if applicable):

          4.16.0-0.nightly-2024-03-13-061822

      How reproducible:

          Always

      Steps to Reproduce:

      1. Create fresh HCP cluster.
      2. Create keycloak as OIDC server exposed as a Route which uses cluster's default ingress certificate as the serving certificate.
      3. Configure clients necessarily on keycloak admin UI.
      4. Configure external OIDC:
      $ oc create configmap keycloak-oidc-ca --from-file=ca-bundle.crt=router-ca/ca.crt --kubeconfig $MGMT_KUBECONFIG -n clusters
      
      $ oc patch hc $HC_NAME -n clusters --kubeconfig $MGMT_KUBECONFIG --type=merge -p="
      spec:
        configuration:
          authentication:
            oidcProviders:
            - claimMappings:
                groups:
                  claim: groups
                  prefix: 'oidc-groups-test:'
                username:
                  claim: email
                  prefixPolicy: Prefix
                  prefix:
                    prefixString: 'oidc-user-test:'
              issuer:
                audiences:
                - $AUDIENCE_1
                - $AUDIENCE_2
                issuerCertificateAuthority:
                  name: keycloak-oidc-ca
                issuerURL: $ISSUER_URL
              name: keycloak-oidc-server
              oidcClients:
              - clientID: $CONSOLE_CLIENT_ID
                clientSecret:
                  name: $CONSOLE_CLIENT_SECRET_NAME
                componentName: console
                componentNamespace: openshift-console
            type: OIDC
      "
      
      5. Check pods should be renewed, but new pod is CrashLoopBackOff:
      $ oc get po -n clusters-$HC_NAME --kubeconfig $MGMT_KUBECONFIG --sort-by metadata.creationTimestamp | tail -n 4
      openshift-apiserver-65f8c5f545-x2vdf                  3/3     Running            0               5h8m
      community-operators-catalog-57dd5886f7-jq25f          1/1     Running            0               4h1m
      kube-apiserver-5d75b5b848-c9c8r                       4/5     CrashLoopBackOff   25 (3m9s ago)   107m
      
      $ oc logs --timestamps -n clusters-$HC_NAME --kubeconfig $MGMT_KUBECONFIG -c kube-apiserver kube-apiserver-5d75b5b848-gk2t8
      ...
      2024-03-18T09:11:14.836540684Z I0318 09:11:14.836495       1 dynamic_cafile_content.go:119] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/certs/client-ca/ca.crt"
      2024-03-18T09:11:14.837725839Z E0318 09:11:14.837695       1 run.go:74] "command failed" err="jwt[0].issuer.certificateAuthority: Invalid value: \"<omitted>\": data does not contain any valid RSA or ECDSA certificates"
      

      Actual results:

      5. New kube-apiserver pod is CrashLoopBackOff.
      
      `oc explain` for issuerCertificateAuthority says the configmap data should use ca-bundle.crt. But I also tried to use ca.crt in configmap's data, got same result.

      Expected results:

      6. No CrashLoopBackOff.

      Additional info:
      Below is my RCA for the CrashLoopBackOff kube-apiserver pod:
      Check if it is valid RSA certificate, it is valid:

      $ openssl x509 -noout -text -in router-ca/ca.crt | grep -i rsa
              Signature Algorithm: sha256WithRSAEncryption
                  Public Key Algorithm: rsaEncryption
          Signature Algorithm: sha256WithRSAEncryption
      

      So, the CA certificate has no issue.
      Above pod logs show "/etc/kubernetes/certs/oidc-ca/ca.crt" is used. Double checked the configmap:

      $ oc get cm auth-config -n clusters-$HC_NAME --kubeconfig $MGMT_KUBECONFIG -o jsonpath='{.data.auth\.json}' | jq | ~/auto/json2yaml.sh
      ---
      kind: AuthenticationConfiguration
      apiVersion: apiserver.config.k8s.io/v1alpha1
      jwt:
      - issuer:
          url: https://keycloak-keycloak.apps..../realms/master
          certificateAuthority: "/etc/kubernetes/certs/oidc-ca/ca.crt"
      ...
      

      Then debug the CrashLoopBackOff pod:

      The used path /etc/kubernetes/certs/oidc-ca/ca.crt does not exist! The correct path should be /etc/kubernetes/certs/oidc-ca/ca-bundle.crt:

      $ oc debug -n clusters-$HC_NAME --kubeconfig $MGMT_KUBECONFIG -c kube-apiserver kube-apiserver-5d75b5b848-gk2t8
      Starting pod/kube-apiserver-5d75b5b848-gk2t8-debug-kpmlf, command was: hyperkube kube-apiserver --openshift-config=/etc/kubernetes/config/config.json -v2 --encryption-provider-config=/etc/kubernetes/secret-encryption/config.yaml
      sh-5.1$ cat /etc/kubernetes/certs/oidc-ca/ca.crt
      cat: /etc/kubernetes/certs/oidc-ca/ca.crt: No such file or directory
      sh-5.1$ ls /etc/kubernetes/certs/oidc-ca/
      ca-bundle.crt
      sh-5.1$ cat /etc/kubernetes/certs/oidc-ca/ca-bundle.crt
      -----BEGIN CERTIFICATE-----
      MIIDPDCCAiSgAwIBAgIIM3E0ckpP750wDQYJKoZIhvcNAQELBQAwJjESMBAGA1UE
      ...
      

            sjenning Seth Jennings
            xxia-1 Xingxing Xia
            Xingxing Xia Xingxing Xia
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated: