Uploaded image for project: 'Red Hat Advanced Cluster Management'
  1. Red Hat Advanced Cluster Management
  2. ACM-12738

Endpointmetrics does not reconcile CMO Config changes

XMLWordPrintable

    • MCO Sprint 26
    • -
    • None

      Description of problem:

      This was noticed when investigating[ this issue|https://issues.redhat.com/browse/OHSS-35716?focusedId=25131120&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-25131120]. Endpointmetrics applies alertmanager config alongside cert related fields in order to enable platform and userworkload alert forwarding to the hub Alertmanager. Endpointmetrics, however, does not trigger a reconciliation based on changes to the CMO config object

      As such, if another process changes the CMO config, we will lose all alert forwarding until endpointmetrics reconciles again. This impacts all ROSA/Managed OpenShift clusters as they intentionally apply changes via a hive syncset every 2hrs to the CMO config object. 

      Version-Release number of selected component (if applicable):

      All versions afaics.

      How reproducible:

      Steps to Reproduce:

      1. Create an OCP cluster, register it to a hub
      2. Note the changes to the CMO cluster-monitoring-config config map in the spoke cluster
      3. Delete the added Additional additionalAlertmanagerConfigs under prometheusK8s in that config map

      Actual results:

      1. Endpointmetrics will not correct the change

      Expected results:

      1. Endpointmetrics should reapply the alertmanager configuration

      Additional info:

      • We need to watch for changes on the cluster-monitoring-config object in endpointmetrics
      • Reconciliation should only be additive - we should not be removing anything from the configuration
      • AdditionalAlertmanagers should be an array that is checked by value (value being the endpoint) to insure if two operators touch the same field, they are not overwritten by appends

              pgough@redhat.com Philip Gough
              mzardab@redhat.com Moad Zardab
              Xiang Yin Xiang Yin
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: