Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-61522

Unexpected DELETE requests by system:serviceaccount:openshift-operators:jaeger-operator to tokenreviews causing 403 Forbidden audit log entries in RHOCP 4

XMLWordPrintable

    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • Important
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Description of problem:

      The jaeger-operator, running in the openshift-operators namespace, is generating large volumes of 403 Forbidden audit events across all 900+ OpenShift clusters. 
      The audit logs show that the operator is attempting to perform a DELETE operation on tokenreviews.authentication.k8s.io, which is not permitted by its service account system:serviceaccount:openshift-operators:jaeger-operator.
      ~~~
      Failure","message":"tokenreviews.authentication.k8s.io \"jaeger-operator-TEST\" is forbidden: User \"system:serviceaccount:openshift-operators:jaeger-operator\" cannot delete resource \"tokenreviews\" in API group \"authentication.k8s.io\" at the cluster scope","reason":"Forbidden","details":{"name":"jaeger-operator-TEST","group":"authentication.k8s.io","kind":"tokenreviews"},"code":403},"requestReceivedTimestamp":"2025-07-04T13:26:34.485347Z","stageTimestamp":"2025-07-04T13:26:34.485801Z","annotations":{"authorization.k8s.io/decision":"forbid","authorization.k8s.io/reason":""}}
      ~~~
      
      This appears to be an unintended behavior, as the tokenreviews resource is typically used for creating authentication review requests, not deleting them. 
      It's likely that the operator is either misusing the API or there is a code issue leading to incorrect HTTP verbs being triggered (e.g., DELETE instead of POST).
      ~~~
      etcd-0.namspkgtd20p.ecs.dyn.nsroot.net {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"f89c9ea3-7806-4bec-80bf-04ce0e777fbd","stage":"ResponseComplete","requestURI":"/apis/authentication.k8s.io/v1/tokenreviews/jaeger-operator-TEST","verb":"delete","user":{"username":"system:serviceaccount:openshift-operators:jaeger-operator","uid":"fddf3ebf-31df-48c5-acd9-88f2ec319b02","groups":["system:serviceaccounts","system:serviceaccounts:openshift-operators","system:authenticated"],"extra":{"authentication.kubernetes.io/pod-name":["jaeger-operator-6688f9474d-47lcg"],"authentication.kubernetes.io/pod-uid":["b80e3949-6fb4-45d5-99b2-11bb5fd7975b"]}},"sourceIPs":["10.167.98.95"],"userAgent":"jaeger-operator/v0.0.0 (linux/amd64) kubernetes/$Format","objectRef":{"resource":"tokenreviews","name":"jaeger-operator-TEST","apiGroup":"authentication.k8s.io","apiVersion":"v1"},"responseStatus":{"metadata":{},"status":"Failure","message":"tokenreviews.authentication.k8s.io \"jaeger-operator-TEST\" is forbidden: User \"system:serviceaccount:openshift-operators:jaeger-operator\" cannot delete resource \"tokenreviews\" in API group \"authentication.k8s.io\" at the cluster scope","reason":"Forbidden","details":{"name":"jaeger-operator-TEST","group":"authentication.k8s.io","kind":"tokenreviews"},"code":403},"requestReceivedTimestamp":"2025-07-04T13:26:34.485347Z","stageTimestamp":"2025-07-04T13:26:34.485801Z","annotations":{"authorization.k8s.io/decision":"forbid","authorization.k8s.io/reason":""}}
      ~~~

      Version-Release number of selected component (if applicable):

       

      Actual results:

      - jaeger-operator is sending DELETE requests to /apis/authentication.k8s.io/v1/tokenreviews/jaeger-operator-TEST.
      - This results in a 403 Forbidden response from the Kubernetes API server.
      - Thousands of audit log entries are generated across all 900+ clusters.
      - These entries are raising red flags in audit reports, creating operational noise and compliance concern.    

      Expected results:

      - The operator should only perform POST operations on tokenreviews for authentication review purposes.
      - No DELETE operation should be attempted on tokenreviews.
      - No 403 Forbidden audit logs should be generated if the operator is correctly scoped and operating as intended.

      Additional info:

      Impact :
      - High operational impact: 1000s of log entries per cluster, affecting audit log size, performance, and clarity.
      - Compliance concern: Unauthorized access attempts are flagged in security reviews.
      - Customer concern: Customers are confused about why the operator is generating these logs and concerned about potential misbehavior or security implications.
      - Support burden: Increases support load ahead of the Jaeger Operator EOL (Nov 3rd), requiring investigation and customer reassurance.
      
      Note : 
      - The jaeger-operator will reach EOL on Nov 3, 2025, but it must still be supported until then.
      - Operator version reports userAgent string: jaeger-operator/v0.0.0, which may indicate a default value or build issue (should be investigated).
      - The operator likely needs a code review to inspect any logic that might be misinterpreting tokenreviews usage or constructing malformed requests.

              Unassigned Unassigned
              rhn-support-sdharma Suruchi Dharma
              None
              None
              None
              None
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: