Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-60418

MSteams webhook receiver in Alertmanager failing to read webhook_url_file secret, with "unmarshal errors" in alertmanager pod logs

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Undefined Undefined
    • None
    • 4.18.z
    • Monitoring
    • None
    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Description of problem:

      • When creating a Microsoft Teams webhook receiver with `webhook_url_file` field pointing to a secret in `/etc/alertmanager/secrets` of alertmanager pods for example:
      receivers:
        - name: msteams-example
          msteams_configs:
            webhook_url_file: /etc/alertmanager/secrets/secret-alertmanager-webhook/teams_webhook_url    
      • This then leads to unmarshal errors in alertmanager pods logs, pointing to the line in the Alertmanager config containing `webhook_url_file` field:
      ts=2025-08-12T13:20:18.602Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/config_out/alertmanager.env.yaml
      ts=2025-08-12T13:20:18.602Z caller=coordinator.go:118 level=error component=configuration msg="Loading configuration file failed" file=/etc/alertmanager/config_out/alertmanager.env.yaml err="yaml: unmarshal errors:\n  line 22: cannot unmarshal !!map into []*config.MSTeamsConfig" 

      In addition, this change is not incorporated into the running Alertmanager config

      Version-Release number of selected component (if applicable):

       

      Alertmanager v0.27.0    

      How reproducible: Easily reproducible

      Steps to Reproduce:

      1. Reproduced on OCP 4.18.10
      2. Create a secret containing the msteams webhook
        $ oc create secret generic secret-alertmanager-webhook \
        --from-literal=teams_webhook_url='<YOUR-MSTEAMS-WEBHOOK-URL-HERE' \
        -n openshift-monitoring
      1. Update cluster-monitoring-config configmap so that the secret is mounted to the alertmanager pods:
        https://docs.redhat.com/en/documentation/openshift_container_platform/4.18/html/monitoring/configuring-core-platform-monitoring#monitoring-configuring-secrets-for-alertmanager_configuring-alerts-and-notifications
      2. Update alertmanager config with receiver and route as described
      receivers:
        - name: msteams-example
          msteams_configs:      
            webhook_url_file: /etc/alertmanager/secrets/secret-alertmanager-webhook/teams_webhook_url
        routes:
          - receiver: msteams-example
            matchers:
              - severity = warning 

      Actual results:
      The following unmarshal error in alertmanager pods logs are observed:

      ts=2025-08-12T13:20:18.602Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/config_out/alertmanager.env.yaml
      ts=2025-08-12T13:20:18.602Z caller=coordinator.go:118 level=error component=configuration msg="Loading configuration file failed" file=/etc/alertmanager/config_out/alertmanager.env.yaml err="yaml: unmarshal errors:\n  line 22: cannot unmarshal !!map into []*config.MSTeamsConfig

      Expected results:

      Successful webhook integration

      Additional info:

      **

      See the following related discussion relating to this:

      https://github.com/prometheus-operator/prometheus-operator/pull/6346#issuecomment-2034841318

       

       

       

       

              jfajersk@redhat.com Jan Fajerski
              rhn-support-ccostell Cormac Costello
              None
              None
              Junqi Zhao Junqi Zhao
              None
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: