Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-25849

PrometheusAdapter and MetricsServer tasks are conflict prone

    XMLWordPrintable

Details

    • Bug
    • Resolution: Unresolved
    • Undefined
    • 4.16.0
    • 4.16
    • Monitoring
    • None
    • No
    • MON Sprint 246, MON Sprint 248
    • 2
    • False
    • Hide

      None

      Show
      None

    Description

      Description of problem:

      The two tasks will always "UPDATE" the underlying "APIService" resource even when no changes are to be made.
      This behavior significantly elevates the likelihood of encountering conflicts, especially during upgrades, with other controllers that are concurrently monitoring the same resources (CA controller e.g.). Moreover, this consumes resources for unnecessary work.
      
      The tasks rely on CreateOrUpdateAPIService: https://github.com/openshift/cluster-monitoring-operator/blob/5e394dd9de305cb6927a23c31b3f9651aa806fb8/pkg/client/client.go#L1725-L1745 which always "UPDATE" the APIService resource.

      Version-Release number of selected component (if applicable):

          

      How reproducible:

      keep a cluster running then take a look at audit logs concerning the APIService v1beta1.metrics.k8s.io, you could use: oc adm must-gather -- /usr/bin/gather_audit_logs

      Steps to Reproduce:

          1.
          2.
          3.
          

      Actual results:

      You would see that every "get" is followed by an "update"
      
      You can also take a look at the code taking care of that: https://github.com/openshift/cluster-monitoring-operator/blob/5e394dd9de305cb6927a23c31b3f9651aa806fb8/pkg/client/client.go#L1725-L1745

      Expected results:

      "Updates" should be avoided if no changes are to be made.

      Additional info:

      it'd be even better if we could avoid the "get"s, but that would be another subject to discuss.

      Attachments

        Issue Links

          Activity

            People

              rh-ee-amrini Ayoub Mrini
              rh-ee-amrini Ayoub Mrini
              Junqi Zhao Junqi Zhao
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated: