OCP logins are failing with Unauthorized error while authentications pods are reporting login succeeded.

 Issue reported on OCP 4.10 and still exists after upgraded to 4.12   

   Haven't reproduced

 Issue: oc login is failing with Unauthorized error using htpasswd user (customer was only having htpasswd user but later we realized the same issue exists for kubeadmin user as well) 

~~~
I0222 08:50:06.388583 2631754 request.go:1181] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"configmaps \"motd\" is forbidden: User \"system:anonymous\" cannot get resource \"configmaps\" in API group \"\" in the namespace \"openshift\"","reason":"Forbidden","details":{"name":"motd","kind":"configmaps"},"code":403}
Login failed (401 Unauthorized)
Verify you have provided correct credentials.
~~~

Action item performed so far:


- Upgraded the cluster to 4.11 as the reported version is EOL

- Reconfigured entire httpasswd authentication but no luck

- We have verified the password which we are using is same as which is present in the secret by following below steps.

~~~
# oc get secret htpass-secret -ojsonpath={.data.htpasswd} -n openshift-config | base64 --decode > users.htpasswd
# htpasswd -v users.htpasswd <username>
# htpasswd -v users.htpasswd ocpadmin
~~~

- Enabled "Debug" log for the authentication operator

- While we are getting Unauthorized error in the command line the autentication pods are reporting authentication succeeded 

~~~
2024-02-29T10:04:53.912095998Z I0229 10:04:53.912072       1 basicauth.go:48] Login with provider "kube:admin" failed for login "user"
2024-02-29T10:04:53.953652531Z I0229 10:04:53.953614       1 htpasswd.go:65] identitymapper: got userIdentityMapping: &groupmapper.UserInfoGroupsWrapper{userInfo:(*user.DefaultInfo)(0xc0010321c0), additionalGroups:sets.String{}}
2024-02-29T10:04:53.953652531Z I0229 10:04:53.953636       1 basicauth.go:51] Login with provider "my_htpasswd_provider" succeeded for login "user": &groupmapper.UserInfoGroupsWrapper{userInfo:(*user.DefaultInfo)(0xc0010321c0), additionalGroups:sets.String{}}
2024-02-29T10:04:53.953652531Z I0229 10:04:53.953645       1 authenticator.go:56] OAuth authentication succeeded: &groupmapper.UserInfoGroupsWrapper{userInfo:(*user.DefaultInfo)(0xc0010321c0), additionalGroups:sets.String{}}
~~~

- We have tried login from multiple nodes including ocp nodes as well as from the authentication operator pods but failing with same error.

- After the first login attempt the identity and users, authaccesstoken is getting created for the user.

- As we seen NTP alert in the cluster, we do suspect that might cause these kind of login issues.

- While looking at the configuration we were not seen any machineconfig for the NTP configuration's, so by default node are taking NTP servers from redhat ntp server pool but the cluster is disconnected hence no synchronization happening.

- To confirm the time synchronization was not the cause, we have enabled internet access to sync with upstream red hat NTP server's. once the internet enabled we have restarted NTP service from all the nodes and nodes are got synchronized.

- Also, we have verified there is no NTP alert exist in the cluster after enabling internet by using below commands

~~~
$ token=`oc sa get-token prometheus-k8s -n openshift-monitoring`

curl -k -H "Authorization: Bearer $token" 'https://alertmanager-main-openshift-monitoring.apps.domain/api/v1/alerts' |  jq '.data[].labels'
~~~

- After fixing the NTP issue, we have reconfigured the htpasswd authentication and deleted all the users and identity which is already exists. but the login is failing with same error.

- As the NTP is not the culprit, customer again revoked the internet access from the nodes.

- To confirm whether the login is only impacted to Htpasswd identity provider we were not having any other identity provider configured, also customer is not having the kubeadmin password.

- As the customer lost kubeadmin password, we had to reset it but after the reset the kubeadmin login also failing with same error while authentication pods are reporting the login is succeeded, please see below logs

~~~
0301 09:47:40.384813       1 basicauth.go:51] Login with provider "kube:admin" succeeded for login "kubeadmin": &user.DefaultInfo{Name:"kube:admin", UID:"thwBGQz38EvJPnxHrQjsCuQEH0oD8KRVywsjLPznOYIo07BkMDscHmPe0MAFPlUAaL4__sEPfdsEUg39sVypwA", Groups:[]string(nil), Extra:map[string][]string(nil)}
W0301 09:47:40.384871       1 context.go:182] Failed to set annotations["authentication.openshift.io/username"] to "kube:admin" for audit:"d0736a0b-d976-40a7-a758-40c260a5d0fe", it has already been set to "kubeadmin"
I0301 09:50:20.962866       1 basicauth.go:51] Login with provider "kube:admin" succeeded for login "kubeadmin": &user.DefaultInfo{Name:"kube:admin", UID:"thwBGQz38EvJPnxHrQjsCuQEH0oD8KRVywsjLPznOYIo07BkMDscHmPe0MAFPlUAaL4__sEPfdsEUg39sVypwA", Groups:[]string(nil), Extra:map[string][]string(nil)}
W0301 09:50:20.962917       1 context.go:182] Failed to set annotations["authentication.openshift.io/username"] to "kube:admin" for audit:"a90517c4-7796-4e49-aa98-c3a2f23821e6", it has already been set to "kubeadmin"
~~~

  not able to login to the cluster

   should be able to login