When issuerCertificateAuthority is set, kube-apiserver gets CrashLoopBackOff

When issuerCertificateAuthority is set, kube-apiserver pod is CrashLoopBackOff.

Tried RCA debugging, found the cause is: the path /etc/kubernetes/certs/oidc-ca/ca.crt is incorrect. The expected path should be /etc/kubernetes/certs/oidc-ca/ca-bundle.crt .    

    4.16.0-0.nightly-2024-03-13-061822

    Always

1. Create fresh HCP cluster.
2. Create keycloak as OIDC server exposed as a Route which uses cluster's default ingress certificate as the serving certificate.
3. Configure clients necessarily on keycloak admin UI.
4. Configure external OIDC:
$ oc create configmap keycloak-oidc-ca --from-file=ca-bundle.crt=router-ca/ca.crt --kubeconfig $MGMT_KUBECONFIG -n clusters

$ oc patch hc $HC_NAME -n clusters --kubeconfig $MGMT_KUBECONFIG --type=merge -p="
spec:
  configuration:
    authentication:
      oidcProviders:
      - claimMappings:
          groups:
            claim: groups
            prefix: 'oidc-groups-test:'
          username:
            claim: email
            prefixPolicy: Prefix
            prefix:
              prefixString: 'oidc-user-test:'
        issuer:
          audiences:
          - $AUDIENCE_1
          - $AUDIENCE_2
          issuerCertificateAuthority:
            name: keycloak-oidc-ca
          issuerURL: $ISSUER_URL
        name: keycloak-oidc-server
        oidcClients:
        - clientID: $CONSOLE_CLIENT_ID
          clientSecret:
            name: $CONSOLE_CLIENT_SECRET_NAME
          componentName: console
          componentNamespace: openshift-console
      type: OIDC
"

5. Check pods should be renewed, but new pod is CrashLoopBackOff:
$ oc get po -n clusters-$HC_NAME --kubeconfig $MGMT_KUBECONFIG --sort-by metadata.creationTimestamp | tail -n 4
openshift-apiserver-65f8c5f545-x2vdf                  3/3     Running            0               5h8m
community-operators-catalog-57dd5886f7-jq25f          1/1     Running            0               4h1m
kube-apiserver-5d75b5b848-c9c8r                       4/5     CrashLoopBackOff   25 (3m9s ago)   107m

$ oc logs --timestamps -n clusters-$HC_NAME --kubeconfig $MGMT_KUBECONFIG -c kube-apiserver kube-apiserver-5d75b5b848-gk2t8
...
2024-03-18T09:11:14.836540684Z I0318 09:11:14.836495       1 dynamic_cafile_content.go:119] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/certs/client-ca/ca.crt"
2024-03-18T09:11:14.837725839Z E0318 09:11:14.837695       1 run.go:74] "command failed" err="jwt[0].issuer.certificateAuthority: Invalid value: \"<omitted>\": data does not contain any valid RSA or ECDSA certificates"

5. New kube-apiserver pod is CrashLoopBackOff.

`oc explain` for issuerCertificateAuthority says the configmap data should use ca-bundle.crt. But I also tried to use ca.crt in configmap's data, got same result.

6. No CrashLoopBackOff.

$ openssl x509 -noout -text -in router-ca/ca.crt | grep -i rsa
        Signature Algorithm: sha256WithRSAEncryption
            Public Key Algorithm: rsaEncryption
    Signature Algorithm: sha256WithRSAEncryption

$ oc get cm auth-config -n clusters-$HC_NAME --kubeconfig $MGMT_KUBECONFIG -o jsonpath='{.data.auth\.json}' | jq | ~/auto/json2yaml.sh
---
kind: AuthenticationConfiguration
apiVersion: apiserver.config.k8s.io/v1alpha1
jwt:
- issuer:
    url: https://keycloak-keycloak.apps..../realms/master
    certificateAuthority: "/etc/kubernetes/certs/oidc-ca/ca.crt"
...

$ oc debug -n clusters-$HC_NAME --kubeconfig $MGMT_KUBECONFIG -c kube-apiserver kube-apiserver-5d75b5b848-gk2t8
Starting pod/kube-apiserver-5d75b5b848-gk2t8-debug-kpmlf, command was: hyperkube kube-apiserver --openshift-config=/etc/kubernetes/config/config.json -v2 --encryption-provider-config=/etc/kubernetes/secret-encryption/config.yaml
sh-5.1$ cat /etc/kubernetes/certs/oidc-ca/ca.crt
cat: /etc/kubernetes/certs/oidc-ca/ca.crt: No such file or directory
sh-5.1$ ls /etc/kubernetes/certs/oidc-ca/
ca-bundle.crt
sh-5.1$ cat /etc/kubernetes/certs/oidc-ca/ca-bundle.crt
-----BEGIN CERTIFICATE-----
MIIDPDCCAiSgAwIBAgIIM3E0ckpP750wDQYJKoZIhvcNAQELBQAwJjESMBAGA1UE
...