Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-27344

prometheus-operator triggering too many "updates"

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Normal Normal
    • None
    • 4.15
    • Monitoring
    • None
    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • Moderate
    • No
    • None
    • None
    • None
    • MON Sprint 276
    • 1
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Description of problem:
      While looking at https://issues.redhat.com/browse/OCPBUGS-26594, spasquie@redhat.com noticed that prom operator triggers 3 updates in a row sometimes for the statefulsets.
      Which doesn't seem normal given that the operator should only update statefulsets when necessary https://github.com/prometheus-operator/prometheus-operator/blob/b2863a485ff7b3d5b885d34cbc759c490dfdc5c2/pkg/prometheus/server/operator.go#L1236-L1273
      Sending 3 updates on the same second timestamp should be investigated:

      05:35:09 [UPDATE][      15.1ms] [200] /apis/apps/v1/namespaces/openshift-monitoring/statefulsets/prometheus-k8s    [system:serviceaccount:openshift-monitoring:prometheus-operator]
      05:35:09 [UPDATE][      15.1ms] [200] /apis/apps/v1/namespaces/openshift-monitoring/statefulsets/prometheus-k8s    [system:serviceaccount:openshift-monitoring:prometheus-operator]
      05:35:09 [UPDATE][     9.651ms] [200] /apis/apps/v1/namespaces/openshift-monitoring/statefulsets/prometheus-k8s    [system:serviceaccount:openshift-monitoring:prometheus-operator]
      05:35:09 [UPDATE][     9.651ms] [200] /apis/apps/v1/namespaces/openshift-monitoring/statefulsets/prometheus-k8s    [system:serviceaccount:openshift-monitoring:prometheus-operator]
      

      The must-gather and audot logs are attached.
      (they belong to https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.15-e2e-metal-ipi-sdn-serial-ipv4/1744941767305728000 if its artifacts are still around)

      Version-Release number of selected component (if applicable):

      
          

      How reproducible:

      
          

      Steps to Reproduce:

          1.
          2.
          3.
          

      Actual results:

      
          

      Expected results:

      
          

      Additional info:

      
          

              rh-ee-amrini Ayoub Mrini
              rh-ee-amrini Ayoub Mrini
              None
              None
              Junqi Zhao Junqi Zhao
              None
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: