Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-60307

Cannot use `webhook_url_file` in Alertmanager msteams webhook receiver

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Undefined Undefined
    • 4.19.0
    • 4.18.z
    • Monitoring
    • None
    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Description of problem:
      The option  to utilize `webhook_url_file` option in alertmanager does not function:

      If a secret is created containing the msteams webhook URL, then mounted to the alertmanager pods, with secret file path included in the Alertmanager msteams receiver for example as follows:

      receivers:
        - name: msteams-example
          msteams_configs:
            - webhook_url_file: /etc/alertmanager/secrets/secret-alertmanager-webhook/teams_webhook_url

      This option is documented upstream:

      https://prometheus.io/docs/alerting/latest/configuration/#msteams_config

      Please note upstream issue:
      https://github.com/prometheus-operator/prometheus-operator/issues/6669

      and Pull Requests:
      https://github.com/prometheus-operator/prometheus-operator/pull/6346

      https://github.com/prometheus/alertmanager/pull/3555

      Version-Release number of selected component (if applicable):

      Alertmanager 0.27.0

      How reproducible:
      Easily reproducible.

          

      Steps to Reproduce:

      1. Reproduced on OCP 4.18.10
      2. Create a secret containing the msteams webhook
        $ oc create secret generic secret-alertmanager-webhook \
        --from-literal=teams_webhook_url='<YOUR-MSTEAMS-WEBHOOK-URL-HERE' \
        -n openshift-monitoring
      1. Update cluster-monitoring-config configmap so that the secret is mounted to the alertmanager pods:
        https://docs.redhat.com/en/documentation/openshift_container_platform/4.18/html/monitoring/configuring-core-platform-monitoring#monitoring-configuring-secrets-for-alertmanager_configuring-alerts-and-notifications
      1. Check that the secret is mounted:
      $ oc -n openshift-monitoring exec alertmanager-main-0 -- ls -l /etc/alertmanager/secrets/
      1. Add the file path to the secret to the receiver in the alertmanager config
        (a) extract alertmanager config
        oc -n openshift-monitoring get secret alertmanager-main --template='{{ index .data "alertmanager.yaml" }}' | base64 --decode > alertmanager.yaml 

        (b) Update config to contain both the msteams receiver, containing the secret's file path under `webhook-url-file` and route referencing the receiver for example

      global:
        http_config:
          proxy_from_environment: true
      inhibit_rules:
        - equal:
            - namespace
            - alertname
          source_matchers:
            - severity = critical
          target_matchers:
            - severity =~ warning|info
        - equal:
            - namespace
            - alertname
          source_matchers:
            - severity = warning
          target_matchers:
            - severity = info
      receivers:
        - name: msteams-example
          msteams_configs:
            - webhook_url_file: /etc/alertmanager/secrets/secret-alertmanager-webhook/teams_webhook_url
        - name: 'null'
      route:
        group_by:
          - namespace
        group_interval: 5m
        group_wait: 30s
        receiver: 'null'
        repeat_interval: 12h
        routes:
          - receiver: msteams-example
            matchers:
              - severity = warning
          - receiver: 'null'
            matchers:
              - alertname = Watchdog
      

      Actual results:
      Alertmanager Pod Logs return following error:

      ts=2025-08-08T14:15:17.940Z caller=coordinator.go:118 level=error component=configuration msg="Loading configuration file failed" file=/etc/alertmanager/config_out/alertmanager.env.yaml err="no msteams webhook URL provided"

      Expected results:

      MSTeams webhook integration should correctly forward any notifications to MSTeams.

      Additional info:
      Customer has expressed concern over this being a security issue, as they wish to avoid pushing sensitive info (i.e. the teams URL) in their config to pubic repository.

              jfajersk@redhat.com Jan Fajerski
              rhn-support-ccostell Cormac Costello
              None
              None
              Junqi Zhao Junqi Zhao
              None
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: