Uploaded image for project: 'Observability Documentation'
  1. Observability Documentation
  2. OBSDOCS-1832

Add a missing information about how Alertmanager routes alerts

    • Icon: Story Story
    • Resolution: Unresolved
    • Icon: Undefined Undefined
    • None
    • None
    • Monitoring
    • 8
    • False
    • Hide

      None

      Show
      None
    • False

      During the testing, I found out about how the Alertmanager routes alerts based on the configuration hierarchy. This information was not anywhere in the docs and due to this my configuration did not work.

      Here is a brief explanation (just a draft, needs to be made clearer):

      If multiple routes on the same level match an alert, Alertmanager routes the alert through the first route based on the order in the configuration.
      For example, Alertmanager with the following configuration routes an alert with `service=example-app` and `severity=critical` labels to the `critical` receiver:
      ----
      # ...
      route:
        # ...
        routes:  # <1>
        - matchers:
          - "severity=critical"
          receiver: critical
        - matchers:
          - "service=example-app"
          routes:  # <2>
          - matchers:
            - "severity=critical"
            receiver: team-frontend-page
      # ...
      ----
      
      <1> Routes on the same level match `severity=critical` and `service=example-app` in this order.
      <2> A sub-level route. Alertmanager evaluates this route only if the alert was matched to the `service=example-app` matcher.

       

      This would be good as a separate concept module in the "key concepts" section, and we can link it to this procedure: https://docs.redhat.com/en/documentation/openshift_container_platform/4.18/html/monitoring/configuring-core-platform-monitoring#configuring-alert-routing-default-platform-alerts_configuring-alerts-and-notifications

      A good way to make it clear what we are talking about would include adding the visualized routing trees with different scenarios:

       

      Routing tree:
      .
      └── default-route  receiver: default
          ├──  {severity="critical"}  receiver: critical
          └── {service="example-app"}  receiver: default
              └── {severity="critical"}  receiver: team-frontend-page

       

      VS

       

      Routing tree:
      .
      └── default-route  receiver: default
          ├── {service="example-app"}  receiver: default
              └── {severity="critical"}  receiver: team-frontend-page
          └── {severity="critical"}  receiver: critical

       

       

              eromanov@redhat.com Eliska Romanova
              eromanov@redhat.com Eliska Romanova
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated: