Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-16814

The web console's JS client is using stale tokens

    • Icon: Bug Bug
    • Resolution: Done-Errata
    • Icon: Critical Critical
    • 4.16.0
    • 4.14
    • Management Console
    • None
    • No
    • False
    • Hide

      None

      Show
      None
    • Release Note Not Required
    • In Progress

      Description of problem:

      Starting OpenShift 4.8 (https://docs.openshift.com/container-platform/4.8/release_notes/ocp-4-8-release-notes.html#ocp-4-8-notable-technical-changes), all pods are getting bound SA tokens.
      
      Currently, instead of expiring the token, we use the `service-account-extend-token-expiration` that extends a bound token validity to 1yr and warns in case of a use of a token that would've otherwise been expired.
      
      We want to disable this behavior in a future OpenShift release, which would break the OpenShift web console.
      

      Version-Release number of selected component (if applicable):

      4.8 - 4.14
      

      How reproducible:

      100%
      

      Steps to Reproduce:

      1. install a fresh cluster
      2. wait ~1hr since console pods were deployed for the token rotation to occur
      3. log in to the console and click around
      4. check the kube-apiserver audit logs events for the "authentication.k8s.io/stale-token" annotation
      

      Actual results:

      many occurrences (I doubt I'll be able to upload a text file so I'll show a few audit events in the first comment.
      

      Expected results:

      The web-console re-reads the SA token regularly so that it never uses an expired token
      

      Additional info:

      In a theoretical case where a console pod lasts for a year, it's going to break and won't be able to authenticate to the kube-apiserver.
      
      We are planning on disallowing the use of stale tokens in a future release and we need to make sure that the core platform is not broken so that the metrics we collect from the clusters in the wild are not polluted.
      

            [OCPBUGS-16814] The web console's JS client is using stale tokens

            Since the problem described in this issue should be resolved in a recent advisory, it has been closed.

            For information on the advisory (Critical: OpenShift Container Platform 4.16.0 bug fix and security update), and where to find the updated files, follow the link below.

            If the solution does not work for you, open a new bug report.
            https://access.redhat.com/errata/RHSA-2024:0041

            Errata Tool added a comment - Since the problem described in this issue should be resolved in a recent advisory, it has been closed. For information on the advisory (Critical: OpenShift Container Platform 4.16.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2024:0041

            YaDan Pei added a comment - - edited

            yeah,  I logged into the console and navigate around, maybe clicks are not enough? anyway I verified the fix is working on 416 with steps below

            redhat@redhatdeMacBook-Pro ~ % oc get clusterversion
            NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
            version   4.16.0-0.nightly-2024-01-21-154905   True        False         4h13m   Cluster version is 4.16.0-0.nightly-2024-01-21-154905
            
            redhat@redhatdeMacBook-Pro ~ % oc get pods -n openshift-console
            NAME                         READY   STATUS    RESTARTS   AGE
            console-6c966fb75b-7dfps     1/1     Running   0          3h55m
            console-6c966fb75b-wgwnm     1/1     Running   0          3h55m
            downloads-5bcd554dbf-74mzt   1/1     Running   0          4h5m
            downloads-5bcd554dbf-cmlds   1/1     Running   0          4h5m
            redhat@redhatdeMacBook-Pro ~ % ./check-stale-token.sh 
            
            redhat@redhatdeMacBook-Pro ~ % oc get pods -n openshift-console
            NAME                         READY   STATUS    RESTARTS   AGE
            console-6c966fb75b-7dfps     1/1     Running   0          4h54m
            console-6c966fb75b-wgwnm     1/1     Running   0          4h54m
            downloads-5bcd554dbf-74mzt   1/1     Running   0          5h4m
            downloads-5bcd554dbf-cmlds   1/1     Running   0          5h4m
            redhat@redhatdeMacBook-Pro ~ % ./check-stale-token.sh  
                    
            redhat@redhatdeMacBook-Pro ~ % oc get pods -n openshift-console
            NAME                         READY   STATUS    RESTARTS   AGE
            console-6c966fb75b-7dfps     1/1     Running   0          6h56m
            console-6c966fb75b-wgwnm     1/1     Running   0          6h56m
            downloads-5bcd554dbf-74mzt   1/1     Running   0          7h6m
            downloads-5bcd554dbf-cmlds   1/1     Running   0          7h6m
            redhat@redhatdeMacBook-Pro ~ % ./check-stale-token.sh  

            since  the reported issue is fixed, moving the bug to verified

            YaDan Pei added a comment - - edited yeah,  I logged into the console and navigate around, maybe clicks are not enough? anyway I verified the fix is working on 416 with steps below redhat@redhatdeMacBook-Pro ~ % oc get clusterversion NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS version   4.16.0-0.nightly-2024-01-21-154905   True        False         4h13m   Cluster version is 4.16.0-0.nightly-2024-01-21-154905 redhat@redhatdeMacBook-Pro ~ % oc get pods -n openshift-console NAME                         READY   STATUS    RESTARTS   AGE console-6c966fb75b-7dfps     1/1     Running   0          3h55m console-6c966fb75b-wgwnm     1/1     Running   0          3h55m downloads-5bcd554dbf-74mzt   1/1     Running   0          4h5m downloads-5bcd554dbf-cmlds   1/1     Running   0          4h5m redhat@redhatdeMacBook-Pro ~ % ./check-stale-token.sh  redhat@redhatdeMacBook-Pro ~ % oc get pods -n openshift-console NAME                         READY   STATUS    RESTARTS   AGE console-6c966fb75b-7dfps     1/1     Running   0          4h54m console-6c966fb75b-wgwnm     1/1     Running   0          4h54m downloads-5bcd554dbf-74mzt   1/1     Running   0          5h4m downloads-5bcd554dbf-cmlds   1/1     Running   0          5h4m redhat@redhatdeMacBook-Pro ~ % ./check-stale-token.sh           redhat@redhatdeMacBook-Pro ~ % oc get pods -n openshift-console NAME                         READY   STATUS    RESTARTS   AGE console-6c966fb75b-7dfps     1/1     Running   0          6h56m console-6c966fb75b-wgwnm     1/1     Running   0          6h56m downloads-5bcd554dbf-74mzt   1/1     Running   0          7h6m downloads-5bcd554dbf-cmlds   1/1     Running   0          7h6m redhat@redhatdeMacBook-Pro ~ % ./check-stale-token.sh  since  the reported issue is fixed, moving the bug to verified

            Note that a part of the reproducer is also logging in to the console and having the javascript do a bunch of queries for you. Maybe that could make it work faster a bit.

            Stanislav Láznička (Inactive) added a comment - Note that a part of the reproducer is also logging in to the console and having the javascript do a bunch of queries for you. Maybe that could make it work faster a bit.

            YaDan Pei added a comment - - edited

            I'm able to reproduce  after 6hrs

            redhat@redhatdeMacBook-Pro ~ % oc get  pods -n openshift-console
            NAME                        READY   STATUS    RESTARTS   AGE
            console-5dc4b478df-ftksz    1/1     Running   0          6h56m
            console-5dc4b478df-nvw5c    1/1     Running   0          6h56m
            downloads-98d4948d6-rkqgf   1/1     Running   0          7h
            downloads-98d4948d6-t4mfn   1/1     Running   0          7h 
            
            redhat@redhatdeMacBook-Pro ~ % ./check-stale-token.sh          
            {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"ed20abf8-9ffd-49fe-90ad-0e787457dc47","stage":"ResponseComplete","requestURI":"/apis/console.openshift.io/v1/consoleplugins","verb":"list","user":{"username":"system:serviceaccount:openshift-console:console","uid":"be4b6573-0aab-40f0-872e-7e36586eec87","groups":["system:serviceaccounts","system:serviceaccounts:openshift-console","system:authenticated"],"extra":{"authentication.kubernetes.io/pod-name":["console-5dc4b478df-ftksz"],"authentication.kubernetes.io/pod-uid":["9fbfb692-e6e7-4eb8-aa4c-5cb3a7f29d67"]}},"sourceIPs":["10.0.79.243"],"userAgent":"bridge/v0.0.0 (linux/amd64) kubernetes/$Format","objectRef":{"resource":"consoleplugins","apiGroup":"console.openshift.io","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2024-01-23T06:16:54.179804Z","stageTimestamp":"2024-01-23T06:16:54.191980Z","annotations":{"authentication.k8s.io/stale-token":"subject: system:serviceaccount:openshift-console:console, seconds after warning threshold: 18087","authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"console-extensions-reader\" of ClusterRole \"console-extensions-reader\" to Group \"system:authenticated\""}}
            {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"a07ced18-fa18-478d-9e96-53f52893bca6","stage":"ResponseComplete","requestURI":"/apis/rbac.authorization.k8s.io/v1/namespaces/openshift-console-user-settings/rolebindings","verb":"list","user":{"username":"system:serviceaccount:openshift-console:console","uid":"be4b6573-0aab-40f0-872e-7e36586eec87","groups":["system:serviceaccounts","system:serviceaccounts:openshift-console","system:authenticated"],"extra":{"authentication.kubernetes.io/pod-name":["console-5dc4b478df-ftksz"],"authentication.kubernetes.io/pod-uid":["9fbfb692-e6e7-4eb8-aa4c-5cb3a7f29d67"]}},"sourceIPs":["10.0.79.243"],"userAgent":"bridge/v0.0.0 (linux/amd64) kubernetes/$Format","objectRef":{"resource":"rolebindings","namespace":"openshift-console-user-settings","apiGroup":"rbac.authorization.k8s.io","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2024-01-23T06:16:54.183502Z","stageTimestamp":"2024-01-23T06:16:54.191370Z","annotations":{"authentication.k8s.io/stale-token":"subject: system:serviceaccount:openshift-console:console, seconds after warning threshold: 18087","authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by RoleBinding \"console-user-settings-admin/openshift-console-user-settings\" of Role \"console-user-settings-admin\" to ServiceAccount \"console/openshift-console\""}}
            {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"4181bc3a-cefe-453e-9910-b54ebb4d3b2d","stage":"ResponseComplete","requestURI":"/apis/console.openshift.io/v1/consoleplugins","verb":"list","user":{"username":"system:serviceaccount:openshift-console:console","uid":"be4b6573-0aab-40f0-872e-7e36586eec87","groups":["system:serviceaccounts","system:serviceaccounts:openshift-console","system:authenticated"],"extra":{"authentication.kubernetes.io/pod-name":["console-5dc4b478df-nvw5c"],"authentication.kubernetes.io/pod-uid":["5ceda8b3-215d-442f-b9bd-eb7b049a7693"]}},"sourceIPs":["10.128.0.70"],"userAgent":"bridge/v0.0.0 (linux/amd64) kubernetes/$Format","objectRef":{"resource":"consoleplugins","apiGroup":"console.openshift.io","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2024-01-23T06:16:54.216360Z","stageTimestamp":"2024-01-23T06:16:54.225114Z","annotations":{"authentication.k8s.io/stale-token":"subject: system:serviceaccount:openshift-console:console, seconds after warning threshold: 18087","authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"console-extensions-reader\" of ClusterRole \"console-extensions-reader\" to Group \"system:authenticated\""}}
            {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"0f0fd250-3eec-4244-9c38-1b6d0fd81b23","stage":"ResponseComplete","requestURI":"/apis/rbac.authorization.k8s.io/v1/namespaces/openshift-console-user-settings/rolebindings","verb":"list","user":{"username":"system:serviceaccount:openshift-console:console","uid":"be4b6573-0aab-40f0-872e-7e36586eec87","groups":["system:serviceaccounts","system:serviceaccounts:openshift-console","system:authenticated"],"extra":{"authentication.kubernetes.io/pod-name":["console-5dc4b478df-nvw5c"],"authentication.kubernetes.io/pod-uid":["5ceda8b3-215d-442f-b9bd-eb7b049a7693"]}},"sourceIPs":["10.0.31.103"],"userAgent":"bridge/v0.0.0 (linux/amd64) kubernetes/$Format","objectRef":{"resource":"rolebindings","namespace":"openshift-console-user-settings","apiGroup":"rbac.authorization.k8s.io","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2024-01-23T06:16:54.216578Z","stageTimestamp":"2024-01-23T06:16:54.221070Z","annotations":{"authentication.k8s.io/stale-token":"subject: system:serviceaccount:openshift-console:console, seconds after warning threshold: 18087","authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by RoleBinding \"console-user-settings-admin/openshift-console-user-settings\" of Role \"console-user-settings-admin\" to ServiceAccount \"console/openshift-console\""}}

            YaDan Pei added a comment - - edited I'm able to reproduce  after 6hrs redhat@redhatdeMacBook-Pro ~ % oc get  pods -n openshift-console NAME                        READY   STATUS    RESTARTS   AGE console-5dc4b478df-ftksz    1/1     Running   0          6h56m console-5dc4b478df-nvw5c    1/1     Running   0          6h56m downloads-98d4948d6-rkqgf   1/1     Running   0          7h downloads-98d4948d6-t4mfn   1/1     Running   0          7h redhat@redhatdeMacBook-Pro ~ % ./check-stale-token.sh           { "kind" : "Event" , "apiVersion" : "audit.k8s.io/v1" , "level" : "Metadata" , "auditID" : "ed20abf8-9ffd-49fe-90ad-0e787457dc47" , "stage" : "ResponseComplete" , "requestURI" : "/apis/console.openshift.io/v1/consoleplugins" , "verb" : "list" , "user" :{ "username" : "system:serviceaccount:openshift-console:console" , "uid" : "be4b6573-0aab-40f0-872e-7e36586eec87" , "groups" :[ "system:serviceaccounts" , "system:serviceaccounts:openshift-console" , "system:authenticated" ], "extra" :{ "authentication.kubernetes.io/pod-name" :[ "console-5dc4b478df-ftksz" ], "authentication.kubernetes.io/pod-uid" :[ "9fbfb692-e6e7-4eb8-aa4c-5cb3a7f29d67" ]}}, "sourceIPs" :[ "10.0.79.243" ], "userAgent" : "bridge/v0.0.0 (linux/amd64) kubernetes/$Format" , "objectRef" :{ "resource" : "consoleplugins" , "apiGroup" : "console.openshift.io" , "apiVersion" : "v1" }, "responseStatus" :{ "metadata" :{}, "code" :200}, "requestReceivedTimestamp" : "2024-01-23T06:16:54.179804Z" , "stageTimestamp" : "2024-01-23T06:16:54.191980Z" , "annotations" :{ "authentication.k8s.io/stale-token" : "subject: system:serviceaccount:openshift-console:console, seconds after warning threshold: 18087" , "authorization.k8s.io/decision" : "allow" , "authorization.k8s.io/reason" : "RBAC: allowed by ClusterRoleBinding \" console-extensions-reader\ " of ClusterRole \" console-extensions-reader\ " to Group \" system:authenticated\""}} { "kind" : "Event" , "apiVersion" : "audit.k8s.io/v1" , "level" : "Metadata" , "auditID" : "a07ced18-fa18-478d-9e96-53f52893bca6" , "stage" : "ResponseComplete" , "requestURI" : "/apis/rbac.authorization.k8s.io/v1/namespaces/openshift-console-user-settings/rolebindings" , "verb" : "list" , "user" :{ "username" : "system:serviceaccount:openshift-console:console" , "uid" : "be4b6573-0aab-40f0-872e-7e36586eec87" , "groups" :[ "system:serviceaccounts" , "system:serviceaccounts:openshift-console" , "system:authenticated" ], "extra" :{ "authentication.kubernetes.io/pod-name" :[ "console-5dc4b478df-ftksz" ], "authentication.kubernetes.io/pod-uid" :[ "9fbfb692-e6e7-4eb8-aa4c-5cb3a7f29d67" ]}}, "sourceIPs" :[ "10.0.79.243" ], "userAgent" : "bridge/v0.0.0 (linux/amd64) kubernetes/$Format" , "objectRef" :{ "resource" : "rolebindings" , "namespace" : "openshift-console-user-settings" , "apiGroup" : "rbac.authorization.k8s.io" , "apiVersion" : "v1" }, "responseStatus" :{ "metadata" :{}, "code" :200}, "requestReceivedTimestamp" : "2024-01-23T06:16:54.183502Z" , "stageTimestamp" : "2024-01-23T06:16:54.191370Z" , "annotations" :{ "authentication.k8s.io/stale-token" : "subject: system:serviceaccount:openshift-console:console, seconds after warning threshold: 18087" , "authorization.k8s.io/decision" : "allow" , "authorization.k8s.io/reason" : "RBAC: allowed by RoleBinding \" console-user-settings-admin/openshift-console-user-settings\ " of Role \" console-user-settings-admin\ " to ServiceAccount \" console/openshift-console\""}} { "kind" : "Event" , "apiVersion" : "audit.k8s.io/v1" , "level" : "Metadata" , "auditID" : "4181bc3a-cefe-453e-9910-b54ebb4d3b2d" , "stage" : "ResponseComplete" , "requestURI" : "/apis/console.openshift.io/v1/consoleplugins" , "verb" : "list" , "user" :{ "username" : "system:serviceaccount:openshift-console:console" , "uid" : "be4b6573-0aab-40f0-872e-7e36586eec87" , "groups" :[ "system:serviceaccounts" , "system:serviceaccounts:openshift-console" , "system:authenticated" ], "extra" :{ "authentication.kubernetes.io/pod-name" :[ "console-5dc4b478df-nvw5c" ], "authentication.kubernetes.io/pod-uid" :[ "5ceda8b3-215d-442f-b9bd-eb7b049a7693" ]}}, "sourceIPs" :[ "10.128.0.70" ], "userAgent" : "bridge/v0.0.0 (linux/amd64) kubernetes/$Format" , "objectRef" :{ "resource" : "consoleplugins" , "apiGroup" : "console.openshift.io" , "apiVersion" : "v1" }, "responseStatus" :{ "metadata" :{}, "code" :200}, "requestReceivedTimestamp" : "2024-01-23T06:16:54.216360Z" , "stageTimestamp" : "2024-01-23T06:16:54.225114Z" , "annotations" :{ "authentication.k8s.io/stale-token" : "subject: system:serviceaccount:openshift-console:console, seconds after warning threshold: 18087" , "authorization.k8s.io/decision" : "allow" , "authorization.k8s.io/reason" : "RBAC: allowed by ClusterRoleBinding \" console-extensions-reader\ " of ClusterRole \" console-extensions-reader\ " to Group \" system:authenticated\""}} { "kind" : "Event" , "apiVersion" : "audit.k8s.io/v1" , "level" : "Metadata" , "auditID" : "0f0fd250-3eec-4244-9c38-1b6d0fd81b23" , "stage" : "ResponseComplete" , "requestURI" : "/apis/rbac.authorization.k8s.io/v1/namespaces/openshift-console-user-settings/rolebindings" , "verb" : "list" , "user" :{ "username" : "system:serviceaccount:openshift-console:console" , "uid" : "be4b6573-0aab-40f0-872e-7e36586eec87" , "groups" :[ "system:serviceaccounts" , "system:serviceaccounts:openshift-console" , "system:authenticated" ], "extra" :{ "authentication.kubernetes.io/pod-name" :[ "console-5dc4b478df-nvw5c" ], "authentication.kubernetes.io/pod-uid" :[ "5ceda8b3-215d-442f-b9bd-eb7b049a7693" ]}}, "sourceIPs" :[ "10.0.31.103" ], "userAgent" : "bridge/v0.0.0 (linux/amd64) kubernetes/$Format" , "objectRef" :{ "resource" : "rolebindings" , "namespace" : "openshift-console-user-settings" , "apiGroup" : "rbac.authorization.k8s.io" , "apiVersion" : "v1" }, "responseStatus" :{ "metadata" :{}, "code" :200}, "requestReceivedTimestamp" : "2024-01-23T06:16:54.216578Z" , "stageTimestamp" : "2024-01-23T06:16:54.221070Z" , "annotations" :{ "authentication.k8s.io/stale-token" : "subject: system:serviceaccount:openshift-console:console, seconds after warning threshold: 18087" , "authorization.k8s.io/decision" : "allow" , "authorization.k8s.io/reason" : "RBAC: allowed by RoleBinding \" console-user-settings-admin/openshift-console-user-settings\ " of Role \" console-user-settings-admin\ " to ServiceAccount \" console/openshift-console\""}}

            YaDan Pei added a comment -

            Yeah, I waited about 3 hrs 

            % oc get  clusterversion
            NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
            version   4.14.0-0.nightly-2024-01-18-061723   True        False         171m    Cluster version is 4.14.0-0.nightly-2024-01-18-061723
            
            % oc get  pods -n openshift-console
            NAME                        READY   STATUS    RESTARTS   AGE
            console-5dc4b478df-ftksz    1/1     Running   0          176m
            console-5dc4b478df-nvw5c    1/1     Running   0          176m
            downloads-98d4948d6-rkqgf   1/1     Running   0          3h
            downloads-98d4948d6-t4mfn   1/1     Running   0          3h 
            
            % ./check-stale-token.sh      still return nothing

            YaDan Pei added a comment - Yeah, I waited about 3 hrs  % oc get  clusterversion NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS version   4.14.0-0.nightly-2024-01-18-061723   True        False         171m    Cluster version is 4.14.0-0.nightly-2024-01-18-061723 % oc get  pods -n openshift-console NAME                        READY   STATUS    RESTARTS   AGE console-5dc4b478df-ftksz    1/1     Running   0          176m console-5dc4b478df-nvw5c    1/1     Running   0          176m downloads-98d4948d6-rkqgf   1/1     Running   0          3h downloads-98d4948d6-t4mfn   1/1     Running   0          3h % ./check-stale-token.sh still return nothing

            Does `check-stale-token.sh` wait for 2 hrs before it checks the audit-logs?

            Stanislav Láznička (Inactive) added a comment - Does `check-stale-token.sh` wait for 2 hrs before it checks the audit-logs?

            YaDan Pei added a comment -

            slaznick@redhat.com I am not able to reproduce the issue on a 4.14.10 cluster

            here are my reproduce steps:

            $ oc get clusterversion   
            NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
            version   4.14.10   True        False         4h32m   Cluster version is 4.14.10
            
            ==> try to grep 'authentication.k8s.io/stale-token' annotation from kube-apiserver audit logs but got nothing 
            
            $ cat check-stale-token.sh 
            #! /bin/bash
            PATTERN='authentication.k8s.io/stale-token'
            KUBE_APISERVER_PODS=$(oc get po -n openshift-kube-apiserver -l apiserver --no-headers | grep -o "^[^ ]*")
            for i in $KUBE_APISERVER_PODS
            do
              oc rsh -n openshift-kube-apiserver -c kube-apiserver $i bash -c "grep -hE '$PATTERN' /var/log/kube-apiserver/audit*.log || true"
            done | jq -cs 'sort_by(.requestReceivedTimestamp)' | jq -c '.[]' 
            $ ./check-stale-token.sh      // nothing returned

            YaDan Pei added a comment - slaznick@redhat.com I am not able to reproduce the issue on a 4.14.10 cluster here are my reproduce steps: $ oc get clusterversion    NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS version   4.14.10   True        False         4h32m   Cluster version is 4.14.10 ==> try to grep 'authentication.k8s.io/stale-token' annotation from kube-apiserver audit logs but got nothing $ cat check-stale-token.sh  #! /bin/bash PATTERN= 'authentication.k8s.io/stale-token' KUBE_APISERVER_PODS=$(oc get po -n openshift-kube-apiserver -l apiserver --no-headers | grep -o "^[^ ]*" ) for i in $KUBE_APISERVER_PODS do   oc rsh -n openshift-kube-apiserver -c kube-apiserver $i bash -c "grep -hE '$PATTERN' / var /log/kube-apiserver/audit*.log || true " done | jq -cs 'sort_by(.requestReceivedTimestamp)' | jq -c '.[]' $ ./check-stale-token.sh // nothing returned

            Hi jhadvig@redhat.com,

            Bugs should not be moved to Verified without first providing a Release Note Type("Bug Fix" or "No Doc Update") and for type "Bug Fix" the Release Note Text must also be provided. Please populate the necessary fields before moving the Bug to Verified.

            OpenShift Jira Bot added a comment - Hi jhadvig@redhat.com , Bugs should not be moved to Verified without first providing a Release Note Type("Bug Fix" or "No Doc Update") and for type "Bug Fix" the Release Note Text must also be provided. Please populate the necessary fields before moving the Bug to Verified.

            Looks like this bug is far enough along in the workflow that a code fix is ready. Customers and support need to know the backport plan. Please complete the "Target Backport Versions" field to indicate which version(s) will receive the fix.

            OpenShift Jira Bot added a comment - Looks like this bug is far enough along in the workflow that a code fix is ready. Customers and support need to know the backport plan. Please complete the " Target Backport Versions " field to indicate which version(s) will receive the fix.

             

            cat audit_logs.stale_tokens.log2_sanitized | jq -c -r 'select(.annotations["authentication.k8s.io/stale-token"] != null)'
            {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"4b7e70ad-f455-475c-8022-d313532bf23c","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/openshift-console-user-settings/configmaps/user-settings-kubeadmin","verb":"get","user":{"username":"system:serviceaccount:openshift-console:console","uid":"5372852f-ffa1-4388-a904-892c2cd3926b","groups":["system:serviceaccounts","system:serviceaccounts:openshift-console","system:authenticated"],"extra":{"authentication.kubernetes.io/pod-name":["console-b578694cc-r6kl5"],"authentication.kubernetes.io/pod-uid":["b062322a-b6ea-41b2-94e2-7a43866f6479"]}},"sourceIPs":["10.0.66.26"],"userAgent":"bridge/v0.0.0 (linux/amd64) kubernetes/$Format","objectRef":{"resource":"configmaps","namespace":"openshift-console-user-settings","name":"user-settings-kubeadmin","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2023-07-26T13:21:59.983360Z","stageTimestamp":"2023-07-26T13:21:59.984766Z","annotations":{"authentication.k8s.io/stale-token":"subject: system:serviceaccount:openshift-console:console, seconds after warning threshold: 2854","authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by RoleBinding \"console-user-settings-admin/openshift-console-user-settings\" of Role \"console-user-settings-admin\" to ServiceAccount \"console/openshift-console\""}}
            {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"64a48dbd-2612-4813-9125-f8e15abb2866","stage":"ResponseComplete","requestURI":"/apis/rbac.authorization.k8s.io/v1/namespaces/openshift-console-user-settings/roles","verb":"create","user":{"username":"system:serviceaccount:openshift-console:console","uid":"5372852f-ffa1-4388-a904-892c2cd3926b","groups":["system:serviceaccounts","system:serviceaccounts:openshift-console","system:authenticated"],"extra":{"authentication.kubernetes.io/pod-name":["console-b578694cc-r6kl5"],"authentication.kubernetes.io/pod-uid":["b062322a-b6ea-41b2-94e2-7a43866f6479"]}},"sourceIPs":["10.0.66.26"],"userAgent":"bridge/v0.0.0 (linux/amd64) kubernetes/$Format","objectRef":{"resource":"roles","namespace":"openshift-console-user-settings","name":"user-settings-kubeadmin-role","apiGroup":"rbac.authorization.k8s.io","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":201},"requestReceivedTimestamp":"2023-07-26T13:21:59.870563Z","stageTimestamp":"2023-07-26T13:21:59.906956Z","annotations":{"authentication.k8s.io/stale-token":"subject: system:serviceaccount:openshift-console:console, seconds after warning threshold: 2854","authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by RoleBinding \"console-user-settings-admin/openshift-console-user-settings\" of Role \"console-user-settings-admin\" to ServiceAccount \"console/openshift-console\""}}
            {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"e34d8ad4-676b-410c-abfe-fb6dbbc82de2","stage":"ResponseComplete","requestURI":"/apis/rbac.authorization.k8s.io/v1/namespaces/openshift-console-user-settings/rolebindings","verb":"create","user":{"username":"system:serviceaccount:openshift-console:console","uid":"5372852f-ffa1-4388-a904-892c2cd3926b","groups":["system:serviceaccounts","system:serviceaccounts:openshift-console","system:authenticated"],"extra":{"authentication.kubernetes.io/pod-name":["console-b578694cc-r6kl5"],"authentication.kubernetes.io/pod-uid":["b062322a-b6ea-41b2-94e2-7a43866f6479"]}},"sourceIPs":["10.0.66.26"],"userAgent":"bridge/v0.0.0 (linux/amd64) kubernetes/$Format","objectRef":{"resource":"rolebindings","namespace":"openshift-console-user-settings","name":"user-settings-kubeadmin-rolebinding","apiGroup":"rbac.authorization.k8s.io","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":201},"requestReceivedTimestamp":"2023-07-26T13:21:59.908755Z","stageTimestamp":"2023-07-26T13:21:59.942670Z","annotations":{"authentication.k8s.io/stale-token":"subject: system:serviceaccount:openshift-console:console, seconds after warning threshold: 2854","authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by RoleBinding \"console-user-settings-admin/openshift-console-user-settings\" of Role \"console-user-settings-admin\" to ServiceAccount \"console/openshift-console\""}}
            {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"592e8854-18f7-475a-9d30-a1d906bd59d5","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/openshift-console-user-settings/configmaps","verb":"create","user":{"username":"system:serviceaccount:openshift-console:console","uid":"5372852f-ffa1-4388-a904-892c2cd3926b","groups":["system:serviceaccounts","system:serviceaccounts:openshift-console","system:authenticated"],"extra":{"authentication.kubernetes.io/pod-name":["console-b578694cc-r6kl5"],"authentication.kubernetes.io/pod-uid":["b062322a-b6ea-41b2-94e2-7a43866f6479"]}},"sourceIPs":["10.0.66.26"],"userAgent":"bridge/v0.0.0 (linux/amd64) kubernetes/$Format","objectRef":{"resource":"configmaps","namespace":"openshift-console-user-settings","name":"user-settings-kubeadmin","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":201},"requestReceivedTimestamp":"2023-07-26T13:21:59.943670Z","stageTimestamp":"2023-07-26T13:21:59.947277Z","annotations":{"authentication.k8s.io/stale-token":"subject: system:serviceaccount:openshift-console:console, seconds after warning threshold: 2854","authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by RoleBinding \"console-user-settings-admin/openshift-console-user-settings\" of Role \"console-user-settings-admin\" to ServiceAccount \"console/openshift-console\""}}
            {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"5e929360-db7e-49a0-9d67-e74762ece32c","stage":"ResponseComplete","requestURI":"/apis/rbac.authorization.k8s.io/v1/namespaces/openshift-console-user-settings/roles","verb":"create","user":{"username":"system:serviceaccount:openshift-console:console","uid":"5372852f-ffa1-4388-a904-892c2cd3926b","groups":["system:serviceaccounts","system:serviceaccounts:openshift-console","system:authenticated"],"extra":{"authentication.kubernetes.io/pod-name":["console-b578694cc-r6kl5"],"authentication.kubernetes.io/pod-uid":["b062322a-b6ea-41b2-94e2-7a43866f6479"]}},"sourceIPs":["10.128.0.77"],"userAgent":"bridge/v0.0.0 (linux/amd64) kubernetes/$Format","objectRef":{"resource":"roles","namespace":"openshift-console-user-settings","name":"user-settings-kubeadmin-role","apiGroup":"rbac.authorization.k8s.io","apiVersion":"v1"},"responseStatus":{"metadata":{},"status":"Failure","message":"roles.rbac.authorization.k8s.io \"user-settings-kubeadmin-role\" already exists","reason":"AlreadyExists","details":{"name":"user-settings-kubeadmin-role","group":"rbac.authorization.k8s.io","kind":"roles"},"code":409},"requestReceivedTimestamp":"2023-07-26T13:21:59.870253Z","stageTimestamp":"2023-07-26T13:21:59.914225Z","annotations":{"authentication.k8s.io/stale-token":"subject: system:serviceaccount:openshift-console:console, seconds after warning threshold: 2854","authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by RoleBinding \"console-user-settings-admin/openshift-console-user-settings\" of Role \"console-user-settings-admin\" to ServiceAccount \"console/openshift-console\""}}
            {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"beeee30a-1805-429b-a94f-efdf24c94b6b","stage":"ResponseComplete","requestURI":"/apis/rbac.authorization.k8s.io/v1/namespaces/openshift-console-user-settings/rolebindings","verb":"create","user":{"username":"system:serviceaccount:openshift-console:console","uid":"5372852f-ffa1-4388-a904-892c2cd3926b","groups":["system:serviceaccounts","system:serviceaccounts:openshift-console","system:authenticated"],"extra":{"authentication.kubernetes.io/pod-name":["console-b578694cc-r6kl5"],"authentication.kubernetes.io/pod-uid":["b062322a-b6ea-41b2-94e2-7a43866f6479"]}},"sourceIPs":["10.128.0.77"],"userAgent":"bridge/v0.0.0 (linux/amd64) kubernetes/$Format","objectRef":{"resource":"rolebindings","namespace":"openshift-console-user-settings","name":"user-settings-kubeadmin-rolebinding","apiGroup":"rbac.authorization.k8s.io","apiVersion":"v1"},"responseStatus":{"metadata":{},"status":"Failure","message":"rolebindings.rbac.authorization.k8s.io \"user-settings-kubeadmin-rolebinding\" already exists","reason":"AlreadyExists","details":{"name":"user-settings-kubeadmin-rolebinding","group":"rbac.authorization.k8s.io","kind":"rolebindings"},"code":409},"requestReceivedTimestamp":"2023-07-26T13:21:59.914654Z","stageTimestamp":"2023-07-26T13:21:59.976528Z","annotations":{"authentication.k8s.io/stale-token":"subject: system:serviceaccount:openshift-console:console, seconds after warning threshold: 2854","authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by RoleBinding \"console-user-settings-admin/openshift-console-user-settings\" of Role \"console-user-settings-admin\" to ServiceAccount \"console/openshift-console\""}} 

             

             

            Stanislav Láznička (Inactive) added a comment -   cat audit_logs.stale_tokens.log2_sanitized | jq -c -r 'select(.annotations[ "authentication.k8s.io/stale-token" ] != null )' { "kind" : "Event" , "apiVersion" : "audit.k8s.io/v1" , "level" : "Metadata" , "auditID" : "4b7e70ad-f455-475c-8022-d313532bf23c" , "stage" : "ResponseComplete" , "requestURI" : "/api/v1/namespaces/openshift-console-user-settings/configmaps/user-settings-kubeadmin" , "verb" : "get" , "user" :{ "username" : "system:serviceaccount:openshift-console:console" , "uid" : "5372852f-ffa1-4388-a904-892c2cd3926b" , "groups" :[ "system:serviceaccounts" , "system:serviceaccounts:openshift-console" , "system:authenticated" ], "extra" :{ "authentication.kubernetes.io/pod-name" :[ "console-b578694cc-r6kl5" ], "authentication.kubernetes.io/pod-uid" :[ "b062322a-b6ea-41b2-94e2-7a43866f6479" ]}}, "sourceIPs" :[ "10.0.66.26" ], "userAgent" : "bridge/v0.0.0 (linux/amd64) kubernetes/$Format" , "objectRef" :{ "resource" : "configmaps" , "namespace" : "openshift-console-user-settings" , "name" : "user-settings-kubeadmin" , "apiVersion" : "v1" }, "responseStatus" :{ "metadata" :{}, "code" :200}, "requestReceivedTimestamp" : "2023-07-26T13:21:59.983360Z" , "stageTimestamp" : "2023-07-26T13:21:59.984766Z" , "annotations" :{ "authentication.k8s.io/stale-token" : "subject: system:serviceaccount:openshift-console:console, seconds after warning threshold: 2854" , "authorization.k8s.io/decision" : "allow" , "authorization.k8s.io/reason" : "RBAC: allowed by RoleBinding \" console-user-settings-admin/openshift-console-user-settings\ " of Role \" console-user-settings-admin\ " to ServiceAccount \" console/openshift-console\""}} { "kind" : "Event" , "apiVersion" : "audit.k8s.io/v1" , "level" : "Metadata" , "auditID" : "64a48dbd-2612-4813-9125-f8e15abb2866" , "stage" : "ResponseComplete" , "requestURI" : "/apis/rbac.authorization.k8s.io/v1/namespaces/openshift-console-user-settings/roles" , "verb" : "create" , "user" :{ "username" : "system:serviceaccount:openshift-console:console" , "uid" : "5372852f-ffa1-4388-a904-892c2cd3926b" , "groups" :[ "system:serviceaccounts" , "system:serviceaccounts:openshift-console" , "system:authenticated" ], "extra" :{ "authentication.kubernetes.io/pod-name" :[ "console-b578694cc-r6kl5" ], "authentication.kubernetes.io/pod-uid" :[ "b062322a-b6ea-41b2-94e2-7a43866f6479" ]}}, "sourceIPs" :[ "10.0.66.26" ], "userAgent" : "bridge/v0.0.0 (linux/amd64) kubernetes/$Format" , "objectRef" :{ "resource" : "roles" , "namespace" : "openshift-console-user-settings" , "name" : "user-settings-kubeadmin-role" , "apiGroup" : "rbac.authorization.k8s.io" , "apiVersion" : "v1" }, "responseStatus" :{ "metadata" :{}, "code" :201}, "requestReceivedTimestamp" : "2023-07-26T13:21:59.870563Z" , "stageTimestamp" : "2023-07-26T13:21:59.906956Z" , "annotations" :{ "authentication.k8s.io/stale-token" : "subject: system:serviceaccount:openshift-console:console, seconds after warning threshold: 2854" , "authorization.k8s.io/decision" : "allow" , "authorization.k8s.io/reason" : "RBAC: allowed by RoleBinding \" console-user-settings-admin/openshift-console-user-settings\ " of Role \" console-user-settings-admin\ " to ServiceAccount \" console/openshift-console\""}} { "kind" : "Event" , "apiVersion" : "audit.k8s.io/v1" , "level" : "Metadata" , "auditID" : "e34d8ad4-676b-410c-abfe-fb6dbbc82de2" , "stage" : "ResponseComplete" , "requestURI" : "/apis/rbac.authorization.k8s.io/v1/namespaces/openshift-console-user-settings/rolebindings" , "verb" : "create" , "user" :{ "username" : "system:serviceaccount:openshift-console:console" , "uid" : "5372852f-ffa1-4388-a904-892c2cd3926b" , "groups" :[ "system:serviceaccounts" , "system:serviceaccounts:openshift-console" , "system:authenticated" ], "extra" :{ "authentication.kubernetes.io/pod-name" :[ "console-b578694cc-r6kl5" ], "authentication.kubernetes.io/pod-uid" :[ "b062322a-b6ea-41b2-94e2-7a43866f6479" ]}}, "sourceIPs" :[ "10.0.66.26" ], "userAgent" : "bridge/v0.0.0 (linux/amd64) kubernetes/$Format" , "objectRef" :{ "resource" : "rolebindings" , "namespace" : "openshift-console-user-settings" , "name" : "user-settings-kubeadmin-rolebinding" , "apiGroup" : "rbac.authorization.k8s.io" , "apiVersion" : "v1" }, "responseStatus" :{ "metadata" :{}, "code" :201}, "requestReceivedTimestamp" : "2023-07-26T13:21:59.908755Z" , "stageTimestamp" : "2023-07-26T13:21:59.942670Z" , "annotations" :{ "authentication.k8s.io/stale-token" : "subject: system:serviceaccount:openshift-console:console, seconds after warning threshold: 2854" , "authorization.k8s.io/decision" : "allow" , "authorization.k8s.io/reason" : "RBAC: allowed by RoleBinding \" console-user-settings-admin/openshift-console-user-settings\ " of Role \" console-user-settings-admin\ " to ServiceAccount \" console/openshift-console\""}} { "kind" : "Event" , "apiVersion" : "audit.k8s.io/v1" , "level" : "Metadata" , "auditID" : "592e8854-18f7-475a-9d30-a1d906bd59d5" , "stage" : "ResponseComplete" , "requestURI" : "/api/v1/namespaces/openshift-console-user-settings/configmaps" , "verb" : "create" , "user" :{ "username" : "system:serviceaccount:openshift-console:console" , "uid" : "5372852f-ffa1-4388-a904-892c2cd3926b" , "groups" :[ "system:serviceaccounts" , "system:serviceaccounts:openshift-console" , "system:authenticated" ], "extra" :{ "authentication.kubernetes.io/pod-name" :[ "console-b578694cc-r6kl5" ], "authentication.kubernetes.io/pod-uid" :[ "b062322a-b6ea-41b2-94e2-7a43866f6479" ]}}, "sourceIPs" :[ "10.0.66.26" ], "userAgent" : "bridge/v0.0.0 (linux/amd64) kubernetes/$Format" , "objectRef" :{ "resource" : "configmaps" , "namespace" : "openshift-console-user-settings" , "name" : "user-settings-kubeadmin" , "apiVersion" : "v1" }, "responseStatus" :{ "metadata" :{}, "code" :201}, "requestReceivedTimestamp" : "2023-07-26T13:21:59.943670Z" , "stageTimestamp" : "2023-07-26T13:21:59.947277Z" , "annotations" :{ "authentication.k8s.io/stale-token" : "subject: system:serviceaccount:openshift-console:console, seconds after warning threshold: 2854" , "authorization.k8s.io/decision" : "allow" , "authorization.k8s.io/reason" : "RBAC: allowed by RoleBinding \" console-user-settings-admin/openshift-console-user-settings\ " of Role \" console-user-settings-admin\ " to ServiceAccount \" console/openshift-console\""}} { "kind" : "Event" , "apiVersion" : "audit.k8s.io/v1" , "level" : "Metadata" , "auditID" : "5e929360-db7e-49a0-9d67-e74762ece32c" , "stage" : "ResponseComplete" , "requestURI" : "/apis/rbac.authorization.k8s.io/v1/namespaces/openshift-console-user-settings/roles" , "verb" : "create" , "user" :{ "username" : "system:serviceaccount:openshift-console:console" , "uid" : "5372852f-ffa1-4388-a904-892c2cd3926b" , "groups" :[ "system:serviceaccounts" , "system:serviceaccounts:openshift-console" , "system:authenticated" ], "extra" :{ "authentication.kubernetes.io/pod-name" :[ "console-b578694cc-r6kl5" ], "authentication.kubernetes.io/pod-uid" :[ "b062322a-b6ea-41b2-94e2-7a43866f6479" ]}}, "sourceIPs" :[ "10.128.0.77" ], "userAgent" : "bridge/v0.0.0 (linux/amd64) kubernetes/$Format" , "objectRef" :{ "resource" : "roles" , "namespace" : "openshift-console-user-settings" , "name" : "user-settings-kubeadmin-role" , "apiGroup" : "rbac.authorization.k8s.io" , "apiVersion" : "v1" }, "responseStatus" :{ "metadata" :{}, "status" : "Failure" , "message" : "roles.rbac.authorization.k8s.io \" user-settings-kubeadmin-role\ " already exists" , "reason" : "AlreadyExists" , "details" :{ "name" : "user-settings-kubeadmin-role" , "group" : "rbac.authorization.k8s.io" , "kind" : "roles" }, "code" :409}, "requestReceivedTimestamp" : "2023-07-26T13:21:59.870253Z" , "stageTimestamp" : "2023-07-26T13:21:59.914225Z" , "annotations" :{ "authentication.k8s.io/stale-token" : "subject: system:serviceaccount:openshift-console:console, seconds after warning threshold: 2854" , "authorization.k8s.io/decision" : "allow" , "authorization.k8s.io/reason" : "RBAC: allowed by RoleBinding \" console-user-settings-admin/openshift-console-user-settings\ " of Role \" console-user-settings-admin\ " to ServiceAccount \" console/openshift-console\""}} { "kind" : "Event" , "apiVersion" : "audit.k8s.io/v1" , "level" : "Metadata" , "auditID" : "beeee30a-1805-429b-a94f-efdf24c94b6b" , "stage" : "ResponseComplete" , "requestURI" : "/apis/rbac.authorization.k8s.io/v1/namespaces/openshift-console-user-settings/rolebindings" , "verb" : "create" , "user" :{ "username" : "system:serviceaccount:openshift-console:console" , "uid" : "5372852f-ffa1-4388-a904-892c2cd3926b" , "groups" :[ "system:serviceaccounts" , "system:serviceaccounts:openshift-console" , "system:authenticated" ], "extra" :{ "authentication.kubernetes.io/pod-name" :[ "console-b578694cc-r6kl5" ], "authentication.kubernetes.io/pod-uid" :[ "b062322a-b6ea-41b2-94e2-7a43866f6479" ]}}, "sourceIPs" :[ "10.128.0.77" ], "userAgent" : "bridge/v0.0.0 (linux/amd64) kubernetes/$Format" , "objectRef" :{ "resource" : "rolebindings" , "namespace" : "openshift-console-user-settings" , "name" : "user-settings-kubeadmin-rolebinding" , "apiGroup" : "rbac.authorization.k8s.io" , "apiVersion" : "v1" }, "responseStatus" :{ "metadata" :{}, "status" : "Failure" , "message" : "rolebindings.rbac.authorization.k8s.io \" user-settings-kubeadmin-rolebinding\ " already exists" , "reason" : "AlreadyExists" , "details" :{ "name" : "user-settings-kubeadmin-rolebinding" , "group" : "rbac.authorization.k8s.io" , "kind" : "rolebindings" }, "code" :409}, "requestReceivedTimestamp" : "2023-07-26T13:21:59.914654Z" , "stageTimestamp" : "2023-07-26T13:21:59.976528Z" , "annotations" :{ "authentication.k8s.io/stale-token" : "subject: system:serviceaccount:openshift-console:console, seconds after warning threshold: 2854" , "authorization.k8s.io/decision" : "allow" , "authorization.k8s.io/reason" : "RBAC: allowed by RoleBinding \" console-user-settings-admin/openshift-console-user-settings\ " of Role \" console-user-settings-admin\ " to ServiceAccount \" console/openshift-console\""}}    

              jhadvig@redhat.com Jakub Hadvig
              slaznick@redhat.com Stanislav Láznička (Inactive)
              YaDan Pei YaDan Pei
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated:
                Resolved: