Uploaded image for project: 'Distributed Tracing'
  1. Distributed Tracing
  2. TRACING-6023

TempoStack version upgrade from 0.16.0-2 to 0.19.0-3 creates networkPolicies that break communication with Kubernetes API.

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • None
    • None
    • Tempo
    • None
    • False
    • Hide

      None

      Show
      None
    • False
    • Tracing Sprint # 284 - 3.9
    • Critical

      Description of problem:

      TempoStack version upgrade from 0.16.0-2 to 0.19.0-3 creates networkPolicies that break communication with Kubernetes API and the pods in openshift-tempo-operator namespace remain in crashloopbackoff state.
      
      Once the policies are removed the openshift-tempo-operator namespace pods keep running normal for sometime until the operator again creates the NetworkPolicies back.  
      
      Error logs from the openshift-tempo-operator pod :
      
      {"level":"error","ts":"2026-02-05T17:14:35.378522Z","logger":"setup","msg":"unable to create controller","controller":"TempoStack","error":"failed to get server groups: Get \"https://172.x.x.x:443/api\": dial tcp 172.x.x.x:443: i/o timeout","stacktrace":"github.com/grafana/tempo-operator/cmd/start.start\n\tgithub.com/grafana/tempo-operator/cmd/start/main.go:81\ngithub.com/spf13/cobra.(*Command).execute\n\tgithub.com/spf13/cobra@v1.9.1/command.go:1019\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tgithub.com/spf13/cobra@v1.9.1/command.go:1148\ngithub.com/spf13/cobra.(*Command).Execute\n\tgithub.com/spf13/cobra@v1.9.1/command.go:1071\nmain.main\n\t./main.go:26\nruntime.main\n\truntime/proc.go:285"}

      Version-Release number of selected component (if applicable):

      TempoStack version : 0.19.0-3
      Openshift version : 4.17

      How reproducible:

      Follow the below steps :

      Steps to Reproduce:

          1. Install the tempoStack operator with version 0.16.0-2
          2. Upgrade the tempoStack operator to version 0.19.0-3 and create      required instances for it.
          3. See if 4 networkPolicies are created in the openshift-tempo-operator namespace :
      tempo-operator-deny-all
      tempo-operator-egress-to-apiserver
      tempo-operator-ingress-to-metrics
      tempo-operator-ingress-webhook
          4. Check if the pods are in crashloopbackoff state in the same namespace
          5. Check for the error logs from the openshift-tempo-operator pod.
          6. Delete the networkPolicies from there and check state of pod.

      Actual results:

      The openshift-tempo-operator pod remain in crashloopbackoff state with it unable to communicate to the kubernetes api service. 

      Expected results:

      The openshift-tempo-operator pod should be running even if the netPol are by default created.

      Additional info:

          

              bbongart@redhat.com Benedikt Bongartz
              rhn-support-atpatil Atharva Patil
              Ke Wang Ke Wang
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: