What:

We need to gather the list of recording rules that are of high value to OCP and add tests to ensure that when they are queried, they return data.

We can start with CMO, but it might also be a good idea to investigate with a broader scope. 

Why:

To help avoid regressions when we drop metrics or there are unexpected changes to the label sets from scraped metrics that cause problems on the cluster. Eg alerts not firing, dashboards not displaying etc 

Example BZ that caused dashboard bug

Acceptance Criteria:

1. All key/important recording rules are identified.
2. A test is added to https://github.com/openshift/origin/blob/aa7a0afc405cb6a7ac48ee12f70dec5100be730a/test/extended/prometheus/prometheus.go#L668-L707 for each identified rule