-
Bug
-
Resolution: Done
-
Undefined
-
None
-
None
-
Quality / Stability / Reliability
-
1
-
False
-
-
False
-
-
-
Tracing Sprint # 270
Version of components:
tempo-operator.v0.15.3
Description of the problem:
When a non-multitenant TempoStack instance is created using the same name as an existing multitenant TempoStack instance, the operator erroneously deletes the Gateway ClusterRole and ClusterRoleBinding associated with the multitenant instance. This behavior affects the multitenant TempoStack's authentication and authorisation for Gateway.
Steps to reproduce the issue:
*Install Tempo Operator built off the latest upstream branch and install the OTEL operator.
*Run the multitenancy test with --skip-delete.
chainsaw test --skip-delete --config .chainsaw-openshift.yaml tests/e2e-openshift/multitenancy
*The test passes and the required Gateway ClusterRole and ClusterRoleBinding is present for the TempoStack instance.
% oc get clusterrole | grep -i chainsaw-multitenancy && oc get clusterrolebinding | grep -i chainsaw-multitenancy tempo-simplest-gateway-chainsaw-multitenancy 2025-04-25T06:15:57Z tempo-simplest-gateway-chainsaw-multitenancy ClusterRole/tempo-simplest-gateway-chainsaw-multitenancy
*Now run the route test case with --skip-delete which has the same name as the TempoStack multitenant instance or directly create the CR from the test case.
chainsaw test --skip-delete --config .chainsaw-openshift.yaml tests/e2e-openshift/route
*Check that the Gateway ClusterRole and ClsuterRoleBinding of the multitenant instance is not present.
% oc get clusterrole | grep -i chainsaw-multitenancy && oc get clusterrolebinding | grep -i chainsaw-multitenancy
*In the Jaeger UI and Gateway container logs of the Gateway pod we can see the expected authentication errors when trying to access the Gateway.
Jaeger UI:
{"error":"failed to authenticate redirect request: tokenreviews.authentication.k8s.io is forbidden: User \"system:serviceaccount:chainsaw-multitenancy:tempo-simplest-gateway\" cannot create resource \"tokenreviews\" in API group \"authentication.k8s.io\" at the cluster scope","errorType":"observatorium-api","status":"error"}
{"error":"unauthorized_client","error_description":"The client is not authorized to request a token using this method.","state":"I love Observatorium"}
Gateway container logs:
$ oc logs tempo-simplest-gateway-f8cbddd75-7r42s Defaulted container "tempo-gateway" out of: tempo-gateway, tempo-gateway-opa 2025/04/24 15:52:14 (version=48fde49, branch=main, revision=48fde49) level=info name=observatorium ts=2025-04-24T15:52:14.511167429Z caller=main.go:462 msg="starting observatorium" level=info name=observatorium ts=2025-04-24T15:52:14.512315531Z caller=main.go:557 msg="adding a tenant" tenant=dev level=info name=observatorium ts=2025-04-24T15:52:14.512342828Z caller=main.go:557 msg="adding a tenant" tenant=prod level=info name=observatorium ts=2025-04-24T15:52:14.512747959Z caller=config.go:19 protocol=HTTP msg="enabling server side TLS" level=info name=observatorium ts=2025-04-24T15:52:14.513118177Z caller=api.go:60 msg="gRPC dialing OTel" endpoint=tempo-simplest-distributor.chainsaw-multitenancy.svc.cluster.local:4317 level=info name=observatorium ts=2025-04-24T15:52:14.514434628Z caller=config.go:19 protocol=HTTP msg="enabling server side TLS" level=info name=observatorium ts=2025-04-24T15:52:14.605618771Z caller=main.go:977 msg="starting internal HTTP server" address=0.0.0.0:8081 level=info name=observatorium ts=2025-04-24T15:52:14.606913388Z caller=main.go:873 msg="starting the HTTP server" address=0.0.0.0:8080 level=info name=observatorium ts=2025-04-24T15:52:14.607071852Z caller=main.go:909 msg="starting the gRPC server" address=0.0.0.0:8090 E0424 15:53:29.022370 1 webhook.go:154] Failed to make webhook authenticator request: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:chainsaw-multitenancy:tempo-simplest-gateway" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope E0424 15:53:29.065471 1 webhook.go:154] Failed to make webhook authenticator request: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:chainsaw-multitenancy:tempo-simpl
