WHAT
As RHOAM on Developer Sandbox moves closer to production, we need a way to track multitenant specific metrics. First we need to decide which metrics we want to expose. Below is a list of potentially useful metrics to track; note that the list is not exhaustive and not everything listed may be useful. Additionally, some of these metrics are already created in integreatly-operator

Potentially useful metrics:

• Number of reconciled tenants
• Number of failed tenants and why they failed (i.e. tenant CR created in wrong namespace, user for tenant CR doesn't exist, tenant CR already exists, etc.)
• Number of user signups
• Number of incomplete user signups and why they failed (i.e. end user started account creation but didn't verify email, SMS, etc.)
• Total resource consumption by tenant (API requests, CPU, memory, etc.)
• Total resource consumption from all tenants (API request, CPU, memory, etc.)

HOW
Implement the Requirement metrics, Prom Queries, Alerts, Grafana Dashboards etc

NOTE: There are some metrics we can use out of the box from Sandbox Operators

An interim solution might be a simple script to retrieve some of the above data.

DONE

• Determined which metrics we want to expose
• Implement the required metrics, Prom Queries, Alerts, and Dashboards where applicable