Description of problem:

Sometimes when logging into the web console or refreshing the web browser we will see the kube-apiserver audit log recording a 404 or 403 event for an attempt to get the user-settings-kubeadmin configmap. This is followed shortly after by a request for the expected user-settings-<UID> configmap. The audit log shows the same user making both requests.

For example:

{"level":"Metadata","auditID":"8436f507-8343-4716-a491-d378bd74e0b1","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/openshift-console-user-settings/configmaps/user-settings-kubeadmin","verb":"get","user":{"username":"IAM#jmcmeek@us.ibm.com","uid":"608c6029-7042-447f-9d3d-8a585f2b677a","groups":["system:authenticated:oauth","system:authenticated"],"extra":{"scopes.authorization.openshift.io":["user:full"]}},"sourceIPs":["172.30.73.192","172.30.92.52","10.209.189.30"],"userAgent":"Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:98.0) Gecko/20100101 Firefox/98.0","objectRef":

{"resource":"configmaps","namespace":"openshift-console-user-settings","name":"user-settings-kubeadmin","apiVersion":"v1"}

,"responseStatus":{"metadata":{},"status":"Failure","reason":"NotFound","code":404},"requestReceivedTimestamp":"2022-03-30T22:22:04.709377Z","stageTimestamp":"2022-03-30T22:22:04.718797Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"ibm-admin\" of ClusterRole \"cluster-admin\" to User \"IAM#jmcmeek@us.ibm.com\""}}

{"level":"Metadata","auditID":"09fa7728-7fdf-48e2-9e47-5de47083f053","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/openshift-console-user-settings/configmaps/user-settings-608c6029-7042-447f-9d3d-8a585f2b677a","verb":"get","user":{"username":"IAM#jmcmeek@us.ibm.com","uid":"608c6029-7042-447f-9d3d-8a585f2b677a","groups":["system:authenticated:oauth","system:authenticated"],"extra":{"scopes.authorization.openshift.io":["user:full"]}},"sourceIPs":["172.30.73.192","172.30.92.52","10.209.189.30"],"userAgent":"Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:98.0) Gecko/20100101 Firefox/98.0","objectRef":

{"resource":"configmaps","namespace":"openshift-console-user-settings","name":"user-settings-608c6029-7042-447f-9d3d-8a585f2b677a","apiVersion":"v1"}

,"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2022-03-30T22:22:04.797604Z","stageTimestamp":"2022-03-30T22:22:04.807386Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"ibm-admin\" of ClusterRole \"cluster-admin\" to User \"IAM#jmcmeek@us.ibm.com\""}}

One of or IBM Cloud customers reported this as their security team looks for 403 requests and wanted to understand if this normal behavior or a bug as it looks like the kind of thing that could come from somebody trying to exploit some well-known user.

cvogt was also able to recreate this today: "I was able to reproduce this at times just now when I go to the console when not logged in. There’s a race between the redirect occurring and the web app starting up without a user set."

Version-Release number of selected component (if applicable):

I have recreated this on 4.7.43 and 4.9.24

Customer reported this on 4.7

How reproducible:

Its not consistent. I've seen it sometimes after first logging in, refreshing a browser session, or coming back to the console the next day. Other times the same actions behave as expected.

Steps to Reproduce:
1.
2.
3.

Actual results:

kube-apiserver audit log shows request for user-settings-kubadmin:
{"level":"Metadata","auditID":"8436f507-8343-4716-a491-d378bd74e0b1","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/openshift-console-user-settings/configmaps/user-settings-kubeadmin","verb":"get","user":{"username":"IAM#jmcmeek@us.ibm.com","uid":"608c6029-7042-447f-9d3d-8a585f2b677a","groups":["system:authenticated:oauth","system:authenticated"],"extra":{"scopes.authorization.openshift.io":["user:full"]}},"sourceIPs":["172.30.73.192","172.30.92.52","10.209.189.30"],"userAgent":"Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:98.0) Gecko/20100101 Firefox/98.0","objectRef":

{"resource":"configmaps","namespace":"openshift-console-user-settings","name":"user-settings-kubeadmin","apiVersion":"v1"}

,"responseStatus":{"metadata":{},"status":"Failure","reason":"NotFound","code":404},"requestReceivedTimestamp":"2022-03-30T22:22:04.709377Z","stageTimestamp":"2022-03-30T22:22:04.718797Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"ibm-admin\" of ClusterRole \"cluster-admin\" to User \"IAM#jmcmeek@us.ibm.com\""}}

Expected results:

Web console browser client requests user-settings for the logged in user.

Additional info:

We have an open support case - https://access.redhat.com/support/cases/#/case/03162081 - but support was not able to reproduce this.

Attachments:

• HAR file from doing a refresh of the browser session - runs through initial requests for the correct user-settings configmaps
• audit log records referencing "user-settings". This covers the initial login and some refreshes.