Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-6151

Add `namespace` label automatically to all custom loki logging alerts

XMLWordPrintable

    • False
    • None
    • False
    • NEW
    • NEW
    • Hide
      Before this update, the Loki Operator did not automatically add the default label `namespace` for all alertingrule caused the User-Workload-Monitoring Alertmanager to skip routing such alerts. With this update, all alerting and recording rules acquire the rule namespace as label resolves the issue the Alertmanager routing works again..
      Show
      Before this update, the Loki Operator did not automatically add the default label `namespace` for all alertingrule caused the User-Workload-Monitoring Alertmanager to skip routing such alerts. With this update, all alerting and recording rules acquire the rule namespace as label resolves the issue the Alertmanager routing works again..
    • Bug Fix
    • Log Storage - Sprint 260, Log Storage - Sprint 261
    • Moderate

      Description of problem:

      custom loki logging alerts are not being routed by the default user workload alertmanager config - which by default uses group_by namespace 
      
      customers would need to add to the default config in order to route custom loki logging alerts...but if the label for `namespace` is added then the alerts will be routed by the default config 
      
      ~~~
      $ oc exec -n openshift-user-workload-monitoring alertmanager-user-workload-0 -- amtool config show --alertmanager.url http://localhost:9093 
      global:
        resolve_timeout: 5m
        http_config:
          follow_redirects: true
          enable_http2: true
        smtp_hello: localhost
        smtp_require_tls: true
        pagerduty_url: https://events.pagerduty.com/v2/enqueue
        opsgenie_api_url: https://api.opsgenie.com/
        wechat_api_url: https://qyapi.weixin.qq.com/cgi-bin/
        victorops_api_url: https://alert.victorops.com/integrations/generic/20131114/alert/
        telegram_api_url: https://api.telegram.org
        webex_api_url: https://webexapis.com/v1/messages
      route:
        receiver: Default
        group_by:
        - namespace
        continue: false
      receivers:
      - name: Default
      templates: []
      
      ~~~

      Version-Release number of selected component (if applicable):

          

      How reproducible:

      always   

      Steps to Reproduce:

      enable a separate alertmanager instance for UWL alert routing
      
      ~~~
      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: user-workload-monitoring-config
        namespace: openshift-user-workload-monitoring
      data:
        config.yaml: |
          alertmanager:
            enabled: true 
            enableAlertmanagerConfig: true
      ~~~
      
      Deploy an always firing custom loki logging alert: 
      https://access.redhat.com/articles/7073775 
      
      set up an alert notification endpoint in UWL alertmanager: 
      https://docs.openshift.com/container-platform/4.16/observability/monitoring/managing-alerts.html#configuring-notifications-for-user-defined-alerts_managing-alerts 

      Actual results:

      without the namespace label the notification will not be sent  

      Expected results:

      default config routes custom loki logging alerting notifications in the same way as metric alerts   

      Additional info:

          

              ptsiraki@redhat.com Periklis Tsirakidis
              rhn-support-nigsmith Nigel Smith
              Anping Li Anping Li
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: