Feature Request Overview 

Controller runtime supports the ability to generate metrics for each operator: https://book.kubebuilder.io/reference/metrics

These metrics can be connected to a prometheus server where useful information can be gathered over time to help gauge how the operators themselves are performing.

Business justification

This would streamline the ability to monitor metrics going forward.

We've already had several OOM bugs filed against operators. With customers deploying multiple instances of OSP within the same OCP deployment this could help identify operators with functional and/or performance issues.

Functional requirements

• Ability to connect operator metrics for each operator installed by the openstack-operator. This would involve creation of a ServiceMonitor
• Metrics should be secured
• A decision on what prometheus server to use should be made. It should not be the same one that OpenStack services use as that is a per-namespace solution.
• If the default OpenShift prometheus is used the openstack-operators namespace will need an annotation to enable operators there to be monitored. Additionally, the OCP prometheus should be configured to allow 'user workload monitoring'.
• NOTE: kubebuilder scaffolding uses bearerTokenFile's by default which are not allowed for user workload monitoring. Instead a certificate could be used to store the token used to allow the OCP prometheus to scrape metrics from our operators.

(Optional) Point of contact

• Dan Prince

(Optional) Additional links

https://issues.redhat.com/browse/RHOSSTRAT-983

Example metrics: