At the Thanos Querier level, it is now possible to configure request logging at the query endpoint level. Doing so provides detailed insights about the queries being executed.

The OCP Monitoring documentation should be updated as follows to reflect this new functionality:

• Instruct cluster admins how to optionally enable the query log for Thanos querier.
• The content should mention the typical use cases for enabling the query log and should emphasize that, because there's no support of log rotation, the query log should only be enabled temporarily for troubleshooting purposes to gain insight into queries being executed.

Background

When engineering was investigating BZ 1982302, metrics were quick to identify that one of the Prometheus instances was receiving an abnormal amount of HTTP requests, and the engineering and support teams spent a lot of time trying to identify the originator of the requests. Eventually, it was discovered that most of them originated from a 3rd party tool running periodic queries on the Kube metrics API. Begin able to enable logging temporarily for Thanos Querier would have made this investigation much more straightforward and less time-consuming.