Proposed title of this feature request

Add more indexes to audit logs in Loki

What is the nature and description of the request?

The only indexes currently on audit logs are log_type and kubernetes_hostname. If I am searching logs for the past week, all audit logs need to be downloaded from the S3 store and processed. This is a lot of data to read and parse and will often timeout on large clusters.

Suggest to add indexes on:
objectRef_resource
objectRef_namespace
verb

This query for example

| json | verb="delete"

When looking at a week of data, it will not show the histogram for me, just a timeout error:

Request: /api/proxy/plugin/logging-view-plugin/backend/api/logs/v1/audit/loki/api/v1/query_range?query=sum+by+%28level%29+%28count_over_time%28%7B+log_type%3D%22audit%22+%7D+%7C+json+%7C+verb%3D%22delete%22+%5B2h%5D%29%29&start=1724434818446000000&end=1725039618446000000&step=2h timed out after 30000ms.

Even just reverse sorting on time, to get the oldest logs will timeout as well, with no histogram shown:

Request: /api/proxy/plugin/logging-view-plugin/backend/api/logs/v1/audit/loki/api/v1/query_range?query=%7B+log_type%3D%22audit%22+%7D+%7C+json+%7C+verb%3D%22delete%22&start=1724435177358000000&end=1725039977358000000&limit=100&direction=forward timed out after 30000ms.

Why does the customer need this? (List the business requirements)

We need a reliable way to search and view audit logs to investigate why this happened and when.

List any affected packages or components.

[Please describe]