While investigating BZ 1982302, metrics were quick to identify that one of the Prometheus instances was receiving an abnormal amount of HTTP requests but the engineering and support teams have spent a lot of time trying to identify the originator of the requests. Eventually it was discovered that most of them originated from a 3rd party tool running periodic queries on the Kube metrics API. At the Prometheus level, it is possible to configure a query log file which would provide more insights on the queries being executed (this is similar to the Apache access logs but for the PromQL engine).

DoD:

• Cluster admins can enable the query log for both platform and user-workload Prometheus.
• The option is documented with an emphasis that query log should only be enabled temporarily because there's no support of log rotation.