Uploaded image for project: 'Red Hat Advanced Cluster Management'
  1. Red Hat Advanced Cluster Management
  2. ACM-16518

[ACM-2.11] MCH Observability Operator reconciles too frequent because of MCH

XMLWordPrintable

    • 1
    • False
    • None
    • False
    • Observability Sprint 35, Obs Sprint 36, Observability Sprint 37, Observability Sprint 38
    • Important
    • None

      Description of problem:

      Observability Operator watches the ACM MCH to check if there are changes in the different Container Images. But it means that everytime there is a change on an MCH, it is triggered a reconcile.

      But, MCH would be in an stage of "pending" for different reasons. For example, a wrongly upgrade process, that keeps MCH Pending. But other issues, like any component of MCE failing. When this happens MCH will do updates very much frequent:

      Like several times on a few minutes, or even seconds:

      Every time this happens, it triggers Observability Operator to reconcile all the clusters, because it would have changed container images (that would be not the case). In our case, the hub had hundreds of spokes. So, a Pending MCH, is changing frequently, that is triggering Observability to reconcile all the spoke clusters (hundreds):

      This makes the operator (and cluster) to be exhaust. Flooding logs, and making many APIServer requests.

      It would be improved, to detect if the MCH changes made any difference that should be applied.  For me it is happening on ACM2.10, but I guess it would be happening in other (including newer) versions.

      By the way, this make more likely to appear the bug about the "fatal error map read write".

      Version-Release number of selected component (if applicable):

      How reproducible:

      Steps to Reproduce:

      1.  
      2.  
      3. ...

      Actual results:

      Expected results:

      Additional info:

              rh-ee-coquadro Coleen Iona Quadros
              jgato@redhat.com Jose Gato Luis
              Jose Gato Luis
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: