Description of problem:

When upgrading from OpenShift Logging 5.3 to OpenShift Logging 5.4.4, audit logs are no longer visible in Kibana. This behaviour was reported by a customer in a Support Case and I was able to reproduce the issue internally.

Some audit logs still seem to be shipped ("SERVICE_START" and "SERVICE_STOP" audit entries) but no other audit logs are displayed despite audit logging being set up correctly.

Version-Release number of selected component (if applicable):

• OpenShift Container Platform 4.10.26
• OpenShift Logging 5.4.4 upgraded from OpenShift Logging 5.3.10

How reproducible:

Always

Steps to Reproduce:
1. Install OpenShift Logging 5.3 on an OpenShift Container Platform cluster.
2. Create a ClusterLogging as described in the documentation. Configure audit logs to be shipped by creating the following ClusterLogForwarder:

apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: instance
  namespace: openshift-logging
spec:
  pipelines:
  - inputRefs:
    - infrastructure
    - application
    - audit
    name: enable-default-log-store
    outputRefs:
    - default

3. Go to Kibana, create a "audit*" index and observe that there are hundreds of audit log entries being shipped constantly
4. Upgrade to OpenShift Logging 5.4.4 ("stable-5.4")

Actual results:

Go to Kibana and observe that there are only a few audit log entries being shown

Expected results:

Go to Kibana and observe that all audit log entries are shown

Additional info:

• Will append additional "oc adm inspect" output for both versions to this Bug