The Alerting consistency document has a section on the alert structure [1] (e.g. naming, labels and annotations). These conventions should be enforced in the openshift/origin e2e tests.

• severity label should be info, warning or critical
• alert name should be camel-case
• description and summary annotations are present
• critical alerts have a runbook_url annotation pointing to a valid link.

DoD

• Tests in openshift/origin that avoid shipping "invalid" alerting rules in OCP. Existing alerts which aren't compliant have an exception in the tests.
• Bugzillas created for not-compliant alerts so teams are aware and can fix it before 4.10.

[1] https://github.com/openshift/enhancements/blob/master/enhancements/monitoring/alerting-consistency.md#documentation-required