Epic Goal

Currently when we forward alerts from spokes to the Hub alertmanager, the route points to the alertmanager service, that will round robin requests to all alertmanager replicas depending on the number replicas defined. Alertmanager notification forwards are expected to be forward to all (best attempt) replicas, with the replicas gossiping on which alertmanager should forward the notification to avoid duplicates. 

Because we round-robin this request, the notification request will never hit more than one replica, and thus the notification upstream will be sent more than once (e.g. {source=alertmanager-1} and {source=alertmanager-2} in a slack notification. 

Why is this important?

This can cause duplicate notifications on the same alert. 

Scenarios

...

Acceptance Criteria

Dependencies (internal and external)

1. ...

Previous Work (Optional):

1. ...

Open questions:

1. …

Done Checklist

• CI - CI is running, tests are automated and merged.
• Release Enablement <link to Feature Enablement Presentation>
• DEV - Upstream code and tests merged: <link to meaningful PR or GitHub
  Issue>
• DEV - Upstream documentation merged: <link to meaningful PR or GitHub
  Issue>
• DEV - Downstream build attached to advisory: <link to errata>
• QE - Test plans in Polarion: <link or reference to Polarion>
• QE - Automated tests merged: <link or reference to automated tests>
• DOC - Doc issue opened with a completed template. Separate doc issue
  opened for any deprecation, removal, or any current known
  issue/troubleshooting removal from the doc, if applicable.