Uploaded image for project: 'Red Hat Advanced Cluster Management'
  1. Red Hat Advanced Cluster Management
  2. ACM-12738

Endpointmetrics does not reconcile CMO Config changes

XMLWordPrintable

    • MCO Sprint 26
    • -
    • None

      Description of problem:

      This was noticed when investigating[ this issue|https://issues.redhat.com/browse/OHSS-35716?focusedId=25131120&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-25131120]. Endpointmetrics applies alertmanager config alongside cert related fields in order to enable platform and userworkload alert forwarding to the hub Alertmanager. Endpointmetrics, however, does not trigger a reconciliation based on changes to the CMO config object

      As such, if another process changes the CMO config, we will lose all alert forwarding until endpointmetrics reconciles again. This impacts all ROSA/Managed OpenShift clusters as they intentionally apply changes via a hive syncset every 2hrs to the CMO config object. 

      Version-Release number of selected component (if applicable):

      All versions afaics.

      How reproducible:

      Steps to Reproduce:

      1. Create an OCP cluster, register it to a hub
      2. Note the changes to the CMO cluster-monitoring-config config map in the spoke cluster
      3. Delete the added Additional additionalAlertmanagerConfigs under prometheusK8s in that config map

      Actual results:

      1. Endpointmetrics will not correct the change

      Expected results:

      1. Endpointmetrics should reapply the alertmanager configuration

      Additional info:

      • We need to watch for changes on the cluster-monitoring-config object in endpointmetrics
      • Reconciliation should only be additive - we should not be removing anything from the configuration
      • AdditionalAlertmanagers should be an array that is checked by value (value being the endpoint) to insure if two operators touch the same field, they are not overwritten by appends

            pgough@redhat.com Philip Gough
            mzardab@redhat.com Moad Zardab
            Xiang Yin Xiang Yin
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: