[Seems a release blocker] 4.14 nightly HyperShift hosted cluster aws-pod-identity does not work

4.14.0-0.nightly-2023-08-11-055332

Always

1.
$ export KUBECONFIG=/path/to/hypershift-hosted-cluster/kubeconfig
$ ogcv
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.14.0-0.nightly-2023-08-11-055332   True        False         8h      Cluster version is 4.14.0-0.nightly-2023-08-11-055332
$ oc get mutatingwebhookconfigurations --context admin
NAME               WEBHOOKS   AGE
aws-pod-identity   1          6h5m

$ oc get --raw=/.well-known/openid-configuration | jq -r '.issuer'
https://xxxx.s3.us-east-2.amazonaws.com/hypershift-xxxx

2.
$ oc new-project xxia-proj
$ oc create sa aws-provider
serviceaccount/aws-provider created

3.
$ ccoctl aws create-iam-roles --name=xxia --region=$REGION --credentials-requests-dir=credentialsrequest-dir-aws --identity-provider-arn=arn:aws:iam::xxxx:oidc-provider/xxxx.s3.us-east-2.amazonaws.com/hypershift-xxxx --output-dir=credrequests-ccoctl-output
2023/08/24 17:54:32 Role arn:aws:iam::xxxx:role/xxia-xxia-proj-aws-creds created
2023/08/24 17:54:32 Saved credentials configuration to: credrequests-ccoctl-output/manifests/xxia-proj-aws-creds-credentials.yaml
2023/08/24 17:54:32 Updated Role policy for Role xxia-xxia-proj-aws-creds

4.
$ oc annotate sa/aws-provider eks.amazonaws.com/role-arn="arn:aws:iam::xxxx:role/xxia-xxia-proj-aws-creds"
$ oc create deployment aws-cli --image=amazon/aws-cli --dry-run=client -o yaml -- sleep 360d | sed "/containers/i \      serviceAccountName: aws-provider" | oc create -f -
deployment.apps/aws-cli created
$ oc get po
NAME                               READY   STATUS              RESTARTS   AGE
aws-cli-5c4f6d7d5b-g6d5v           1/1     Running             0          18s

5.
$ oc rsh aws-cli-5c4f6d7d5b-g6d5v
sh-4.2$ env | grep AWS
sh-4.2$ ls /var/run/secrets/eks.amazonaws.com/serviceaccount/token
ls: cannot access /var/run/secrets/eks.amazonaws.com/serviceaccount/token: No such file or directory
sh-4.2$ exit
command terminated with exit code 1

5. No AWS env vars.

5. Should have AWS env vars.

1.
$ ogcv    
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.13.0-0.nightly-2023-08-11-101506   True        False         10h     Cluster version is 4.13.0-0.nightly-2023-08-11-101506
$ oc get --raw=/.well-known/openid-configuration | jq -r '.issuer'
https://aos-xxxx.s3.us-east-2.amazonaws.com/xxxx
$ oc get no                       
NAME                                        STATUS   ROLES    AGE   VERSION
ip-10-0-139-76.us-east-2.compute.internal   Ready    worker   10h   v1.26.6+6bf3f75
...
$ REGION=us-east-2

2.
$ oc new-project xxia-proj
$ oc create sa aws-provider

3.
$ ccoctl aws create-iam-roles --name=xxia-test --region=$REGION --credentials-requests-dir=credentialsrequest-dir-aws --identity-provider-arn=arn:aws:iam::xxxx:oidc-provider/aos-xxxx.s3.us-east-2.amazonaws.com/xxxx --output-dir=credrequests-ccoctl-output
2023/08/24 20:06:53 Role arn:aws:iam::xxxx:role/xxia-test-xxia-proj-aws-creds created 
2023/08/24 20:06:53 Saved credentials configuration to: credrequests-ccoctl-output/manifests/xxia-proj-aws-creds-credentials.yaml
2023/08/24 20:06:53 Updated Role policy for Role xxia-test-xxia-proj-aws-creds

4.
$ oc annotate sa/aws-provider eks.amazonaws.com/role-arn="arn:aws:iam::xxxx:role/xxia-test-xxia-proj-aws-creds"
$ oc create deployment aws-cli --image=amazon/aws-cli --dry-run=client -o yaml -- sleep 360d | sed "/containers/i \      serviceAccountName: aws-provider" | oc create -f -
$ oc get pod               
NAME                       READY   STATUS    RESTARTS   AGE
aws-cli-84875995cc-svszl   1/1     Running   0          16s

5.
$ oc rsh aws-cli-84875995cc-svszl
sh-4.2$ env | grep AWS
AWS_ROLE_ARN=arn:aws:iam::xxxx:role/xxia-test-xxia-proj-aws-creds
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token
AWS_DEFAULT_REGION=us-east-2
AWS_REGION=us-east-2