Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-17113

Monitoring should only update Prometheus resources when there are changes

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Normal Normal
    • None
    • 4.14
    • Monitoring
    • Moderate
    • No
    • 8
    • MON Sprint 245, MON Sprint 249, MON Sprint 256, MON Sprint 258, MON Sprint 259
    • 5
    • False
    • Hide

      None

      Show
      None

      Description of problem

      The 4.14.0-ec.4 monitoring operator makes Update calls for its Prometheus resource about once a minute, even when there are no changes being requested.

      Version-Release number of selected component (if applicable):

      At least 4.14.0-ec.4. I have not looked in other versions.

      How reproducible:

      Seen in build02 and confirmed in the first ec.4 CI run I checked, so likely very reproduceable.

      Steps to Reproduce

      1. Run ec.4 CI.
      2. Check Kube API server audit logs:

      $ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-nightly-4.14-e2e-aws-ovn-serial/1685027676433158144/artifacts/e2e-aws-ovn-serial/gather-audit-logs/artifacts/audit-logs.tar | tar -xz --strip-components=2
$ zgrep -h '"resource":"prometheuses"' kube-apiserver/*.log.gz | jq -r 'select(.verb == "update" and .objectRef.subresource != "status") | .stageTimestamp + " " + (.responseStatus.code | tostring) + " " + .user.username' | sort

      Actual results

      2023-07-28T21:10:30.455712Z 200 system:serviceaccount:openshift-monitoring:cluster-monitoring-operator
2023-07-28T21:11:39.629004Z 200 system:serviceaccount:openshift-monitoring:cluster-monitoring-operator
2023-07-28T21:11:58.727870Z 200 system:serviceaccount:openshift-monitoring:cluster-monitoring-operator
2023-07-28T21:13:24.616877Z 200 system:serviceaccount:openshift-monitoring:cluster-monitoring-operator
2023-07-28T21:13:43.859596Z 200 system:serviceaccount:openshift-monitoring:cluster-monitoring-operator
2023-07-28T21:14:51.770214Z 200 system:serviceaccount:openshift-monitoring:cluster-monitoring-operator
2023-07-28T21:15:10.524179Z 200 system:serviceaccount:openshift-monitoring:cluster-monitoring-operator
...

      Expected results

      Only update calls when a change needs to be pushed to the resource, to save the network and API-server overhead of no-op update calls.

            prasriva@redhat.com Pranshu Srivastava
            trking W. Trevor King
            Junqi Zhao Junqi Zhao
            Jan Fajerski
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated: