Updating oidcProviders does not take effect

Updating oidcProviders does not take effect. See details below.

4.16.0-0.nightly-2024-02-26-155043

Always 

1. Install fresh HCP env and configure external OIDC as steps 1 ~ 4 of https://issues.redhat.com/browse/OCPBUGS-29154 (to avoid repeated typing those steps, only referencing as is here).

2. Pods renewed:
$ oc get po -n clusters-$HC_NAME --kubeconfig $MGMT_KUBECONFIG --sort-by metadata.creationTimestamp
...
network-node-identity-68b7b8dd48-4pvvq               3/3     Running   0          170m
oauth-openshift-57cbd9c797-6hgzx                     2/2     Running   0          170m
kube-controller-manager-66f68c8bd8-tknvc             1/1     Running   0          164m
kube-controller-manager-66f68c8bd8-wb2x9             1/1     Running   0          164m
kube-controller-manager-66f68c8bd8-kwxxj             1/1     Running   0          163m
kube-apiserver-596dcb97f-n5nqn                       5/5     Running   0          29m
kube-apiserver-596dcb97f-7cn9f                       5/5     Running   0          27m
kube-apiserver-596dcb97f-2rskz                       5/5     Running   0          25m
openshift-apiserver-c9455455c-t7prz                  3/3     Running   0          22m
openshift-apiserver-c9455455c-jrwdf                  3/3     Running   0          22m
openshift-apiserver-c9455455c-npvn5                  3/3     Running   0          21m
konnectivity-agent-7bfc7cb9db-bgrsv                  1/1     Running   0          20m
cluster-version-operator-675745c9d6-5mv8m            1/1     Running   0          20m
hosted-cluster-config-operator-559644d45b-4vpkq      1/1     Running   0          20m
konnectivity-agent-7bfc7cb9db-hjqlf                  1/1     Running   0          20m
konnectivity-agent-7bfc7cb9db-gl9b7                  1/1     Running   0          20m

3. oc login can succeed:
$ oc login --exec-plugin=oc-oidc --client-id=$CLIENT_ID --client-secret=$CLIENT_SECRET_VALUE --extra-scopes=email --callback-port=8080
Please visit the following URL in your browser: http://localhost:8080
Logged into "https://a4af9764....elb.ap-southeast-1.amazonaws.com:6443" as "oidc-user-test:xxia@redhat.com" from an external oidc issuer.You don't have any projects. Contact your system administrator to request a project.

4. Update HC by changing claim: email to claim: sub:
$ oc edit hc $HC_NAME -n clusters --kubeconfig $MGMT_KUBECONFIG
...
          username:
            claim: sub
...

Update is picked up:
$ oc get authentication.config cluster -o yaml
...
spec:
  oauthMetadata:
    name: tested-oauth-meta
  oidcProviders:
  - claimMappings:
      groups:
        claim: groups
        prefix: 'oidc-groups-test:'
      username:
        claim: sub
        prefix:
          prefixString: 'oidc-user-test:'
        prefixPolicy: Prefix
    issuer:
      audiences:
      - 76863fb1-xxxxxx
      issuerCertificateAuthority:
        name: ""
      issuerURL: https://login.microsoftonline.com/xxxxxxxx/v2.0
    name: microsoft-entra-id
    oidcClients:
    - clientID: 76863fb1-xxxxxx
      clientSecret:
        name: console-secret
      componentName: console
      componentNamespace: openshift-console
  serviceAccountIssuer: https://xxxxxx.s3.us-east-2.amazonaws.com/hypershift-ci-267402
  type: OIDC
status:
  oidcClients:
  - componentName: console
    componentNamespace: openshift-console
    conditions:
    - lastTransitionTime: "2024-02-28T10:51:17Z"
      message: ""
      reason: OIDCConfigAvailable
      status: "False"
      type: Degraded
    - lastTransitionTime: "2024-02-28T10:51:17Z"
      message: ""
      reason: OIDCConfigAvailable
      status: "False"
      type: Progressing
    - lastTransitionTime: "2024-02-28T10:51:17Z"
      message: ""
      reason: OIDCConfigAvailable
      status: "True"
      type: Available
    currentOIDCClients:
    - clientID: 76863fb1-xxxxxx
      issuerURL: https://login.microsoftonline.com/xxxxxxxx/v2.0
      oidcProviderName: microsoft-entra-id

4. Check pods again:
$ oc get po -n clusters-$HC_NAME --kubeconfig $MGMT_KUBECONFIG --sort-by metadata.creationTimestamp
...
kube-apiserver-596dcb97f-n5nqn                       5/5     Running   0          108m
kube-apiserver-596dcb97f-7cn9f                       5/5     Running   0          106m
kube-apiserver-596dcb97f-2rskz                       5/5     Running   0          104m
openshift-apiserver-c9455455c-t7prz                  3/3     Running   0          102m
openshift-apiserver-c9455455c-jrwdf                  3/3     Running   0          101m
openshift-apiserver-c9455455c-npvn5                  3/3     Running   0          100m
konnectivity-agent-7bfc7cb9db-bgrsv                  1/1     Running   0          100m
cluster-version-operator-675745c9d6-5mv8m            1/1     Running   0          100m
hosted-cluster-config-operator-559644d45b-4vpkq      1/1     Running   0          100m
konnectivity-agent-7bfc7cb9db-hjqlf                  1/1     Running   0          99m
konnectivity-agent-7bfc7cb9db-gl9b7                  1/1     Running   0          99m

No new pods renewed.

5. Check login again, it does not use "sub", still use "email":
$ rm -rf ~/.kube/cache/
$ oc login --exec-plugin=oc-oidc --client-id=$CLIENT_ID --client-secret=$CLIENT_SECRET_VALUE --extra-scopes=email --callback-port=8080
Please visit the following URL in your browser: http://localhost:8080
Logged into "https://xxxxxxx.elb.ap-southeast-1.amazonaws.com:6443" as "oidc-user-test:xxia@redhat.com" from an external oidc issuer.
  
You don't have any projects. Contact your system administrator to request a project.
$ cat ~/.kube/cache/oc/* | jq -r '.id_token' | jq -R 'split(".") | .[] | @base64d | fromjson'
...
{
...
  "email": "xxia@redhat.com",
  "groups": [
...
  ],
...
  "sub": "EEFGfgPXr0YFw_ZbMphFz6UvCwkdFS20MUjDDLdTZ_M",
...

Steps 4 ~ 5: after editing HC field value from "claim: email" to "claim: sub", even if `oc get authentication cluster -o yaml` shows the edited change is propagated:
1> The pods like kube-apiserver are not renewed.
2> After clean-up ~/.kube/cache, `oc login ...` relogin still prints 'Logged into "https://xxxxxxx.elb.ap-southeast-1.amazonaws.com:6443" as "oidc-user-test:xxia@redhat.com" from an external oidc issuer', i.e. still uses old claim "email" as user name, instead of using the new claim "sub".

Steps 4 ~ 5: Pods like kube-apiserver pods should renew after HC editing that changes user claim. The login should print that the new claim is used as user name.