Uploaded image for project: 'Red Hat OpenShift Data Science'
  1. Red Hat OpenShift Data Science
  2. RHODS-1886

E-mail user notifications are not sent when using multiple notification e-mails

XMLWordPrintable

    • IDH Sprint 10, IDH Sprint 11, IDH Sprint 12, IDH Sprint 13, IDH Sprint 14, IDH Sprint 15, IDH Sprint 16, IDH Sprint 17

      Description of problem:

       

      When you install RHODS using the add-on-flow, there is a dialog that lets you configure e-mail notifications for alerts.

       

      if you just add one e-mail address it works fine and e-mails are sent when we fill the notebook PVCs over 90% or 100%

       

      The problem is that if you add multiple e-mail addresses (separated by space, as the dialog recommends) the e-mails are not sent because alertmanager complains about wrong configuration (more info below)

       

      If we look at alertmanager's configmap, we see the list of e-mail addresses:

      - name: 'user-notifications'
        email_configs:
        - to: 'jgarciao@redhat.com jgarciao@gmail.com bdattoma@redhat.com'
          send_resolved: false
          html: '{{ template "email.rhods.html" . }}'
          headers: 
            subject: '{{ template "email.rhods.subject" . }}'
      

       

      After the users PVC is over 90% and the alert is firing in prometheus,  if we look at alertmanager's log, we see that the email with the alert was not sent because a configuration error. It's expecting a coma separated list of e-mail addresses:

      level=error ts=2021-09-17T10:40:03.037Z caller=dispatch.go:309 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="user-notifications/email[0]: notify retry canceled due to unrecoverable error after 1 attempts: parse 'to' addresses: mail: expected comma"
      

       

      Actual results:

      User alarm e-mails are not sent

      Expected results:

      User alarm e-mails to be sent

      Reproducibility (Always/Intermittent/Only Once):

      Always

      Build Details:

      I've tested this in ODS stage with RHODS v1.1.1-21

       

        1. alertmanager-configmap-modified-manually.png
          20 kB
          Jorge Garcia Oncins
        2. prometheus-56f88cbdfb-5x2ds-alertmanager.log
          11 kB
          Jorge Garcia Oncins
        3. rhods-add-on-query-address.png
          70 kB
          Jorge Garcia Oncins
        4. rhods-add-on-query-address-Sep30th.png
          59 kB
          Jorge Garcia Oncins

              mshah10 Maulik Shah (Inactive)
              rhn-support-jgarciao Jorge Garcia Oncins
              Jorge Garcia Oncins Jorge Garcia Oncins
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated:
                Resolved: