Uploaded image for project: 'Observability UI'
  1. Observability UI
  2. OU-662

unable to silence alert from console when userworkload alertmanager is deployed

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Undefined Undefined
    • None
    • OpenShift 4.16, OpenShift 4.18
    • Admin-Console
    • None
    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Alert Silences set in the administrator console are not set against the correct instance when a user workload alertmanger are deployed. 

      replication steps: 

      1: deploy UWL monitoring 

      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: cluster-monitoring-config
        namespace: openshift-monitoring
      data:
        config.yaml: |
          enableUserWorkload: true 

      2: deploy UWL alertmanager: 

       

      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: user-workload-monitoring-config
        namespace: openshift-user-workload-monitoring
      data:
        config.yaml: |
          alertmanager:
            enabled: true 
            enableAlertmanagerConfig: true 

       

      Deploy sample application: 
      https://access.redhat.com/articles/7024800

       

      Query to confirm the alert is firing on the UWL instance: 

      $ oc exec n openshift-user-workload-monitoring alertmanager-user-workload-0 – amtool alert query  -alertmanager.url http://localhost:9093 
      Alertname     Starts At                Summary  State   
      VersionAlert  2025-02-24 20:18:17 UTC           active  

       

      Set a silence in the GUI: label = alertname, value = VersionAlert

      now query the UWL instance for the silence: 

      $ oc exec -n openshift-user-workload-monitoring alertmanager-user-workload-0 – amtool silence query --alertmanager.url http://localhost:9093 
      ID  Matchers  Ends At  Created By  Comment  

      its not there! 
      now query the core instance to find the silence: 

      $ oc -n openshift-monitoring exec -ti alertmanager-main-0 -c alertmanager – amtool silence query --alertmanager.url http://localhost:9093 
      ID                                    Matchers                           Ends At                  Created By  Comment  
      8d1c9189-74f9-40b5-acbb-c5dde5f916e5  alertname="VersionAlert"           2025-02-24 22:22:39 UTC  kube:admin  test     

              gbernal@redhat.com Gabriel Bernal
              rhn-support-nigsmith Nigel Smith
              None
              None
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated: