While investigating BZ 1982302, metrics were quick to identify that one of the Prometheus instances was receiving an abnormal amount of HTTP requests but the engineering and support teams have spent a lot of time trying to identify the originator of the requests. Eventually it was discovered that most of them originated from a 3rd party tool running periodic queries on the Kube metrics API. At the Thanos Querier level, it is possible to configure a request logging at query endpoint level which would provide more insights on the queries being executed.

DoD:

• Cluster admins can enable the query log for Thanos querier.
• The option is documented with an emphasis that query log should only be enabled temporarily because there's no support of log rotation.