Uploaded image for project: 'Docs for Red Hat Developers'
  1. Docs for Red Hat Developers
  2. RHDEVDOCS-3611

Allow OpenShift users to configure request logging for Thanos Querier query endpoint

    XMLWordPrintable

Details

    • devex docs #215 Feb 24-Mar 17, devex docs #216 Mar 17-Apr 7, devex docs #217 Apr 7-Apr 28
    • 5
    • Undefined

    Description

      At the Thanos Querier level, it is now possible to configure request logging at the query endpoint level. Doing so provides detailed insights about the queries being executed.

      The OCP Monitoring documentation should be updated as follows to reflect this new functionality:

      • Instruct cluster admins how to optionally enable the query log for Thanos querier.
      • The content should mention the typical use cases for enabling the query log and should emphasize that, because there's no support of log rotation, the query log should only be enabled temporarily for troubleshooting purposes to gain insight into queries being executed.

      Background

      When engineering was investigating BZ 1982302, metrics were quick to identify that one of the Prometheus instances was receiving an abnormal amount of HTTP requests, and the engineering and support teams spent a lot of time trying to identify the originator of the requests. Eventually, it was discovered that most of them originated from a 3rd party tool running periodic queries on the Kube metrics API. Begin able to enable logging temporarily for Thanos Querier would have made this investigation much more straightforward and less time-consuming.

      Attachments

        Activity

          People

            rhn-support-bburt Brian Burt
            rhn-support-bburt Brian Burt
            Junqi Zhao Junqi Zhao
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: