AU-5 says:

The information system:
a. Alerts [Assignment: organization-defined personnel or roles] in the event of an audit processing failure; and
b. Takes the following additional actions: [Assignment: organization-defined actions to be taken (e.g., shut down information system, overwrite oldest audit records, stop generating audit records)].

a) is easier because the apiserver issues the apiserver_audit_error_total metric. We still need to create a rule that checks for prometheusRule objects that alerts if this metric increases and have remediations that create those prometheusRule objects if they don't exist.

b) is harder because at the moment, the only configuration options related to audit that the apiserver exposes are how big the file can be and how many files to retain. This could be coupled with having a separate partition for the audit log files to prevent running out of space, but doesn't provide any action for other kinds of failures. We need to investigate if it would be enough to leverage AlertManager which can call custom webhooks or whether we need an RFE against the apiservers

Acceptance criteria:

• we create a CaC rule that checks for PrometheusRules that issue alerts based on the apiserver_audit_error_total metric
  • this rule creates the rules as remediations if they don't exist
• decide whether we can cover b) with a webhook through alertmanager or whether we need to file an RFE against the apiservers