RHODS users and admins sometimes need to be alerted (eg. email, in product or other channel) based on some system events. The general idea is they would like to system to proactively notify them so they can take some action. Red Hat has a notification service within cloud.redhat.com. Other services such as Managed Kafka are planning to leverage this service. This epic is for RHODS to utilize this notification service for some initial use cases.

Slides on notification service: https://docs.google.com/presentation/d/17jXLSoSTI9712qOsLkXnTobtTv5oBmEXB2OJzsuBerY/edit#slide=id.g84c55014f8_2_240

Link to Epic Brief : https://docs.google.com/document/d/15QhHS4sYIfU4P4XH0Lg8V30EPW-kRfoRqxTepPStn2U/edit?usp=sharing

Requirements:

1. P0: The system must be able to alert users if the cluster storage is approaching capacity (eg. 80% of capacity, 90% of capacity, 95% of capacity).
2. P0: The system must be able to alert users if a process results in some type of error such as running out of memory. Ex: a long running process runs out of memory at some point - user needs to be alerted as soon as the issue occurs.
3. P1: The system must be able to alert users at the completion of a long running process.
4. P1: The system must be able to alert users based on vCPU consumpiton (eg. 90% of total vCPU capacity).