Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-41602

monitoring-plugin pod generates too many logs

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Undefined Undefined
    • None
    • 4.18
    • Observability UI
    • None
    • None
    • 1
    • False
    • Hide

      None

      Show
      None
    • Hide
      *Cause*: The monitoring-plugin swapped to a different backend which logged each request to it. An additional health check was added to the monitoring-plugin deployment, which causes the `/health` endpoint to get pinged every so often.
      *Consequence*: There is an excess of logs to the `/health` endpoint.
      *Fix*: Remove logging of requests as a default behavior. Add in an additional `log-verbosity` parameter to allow power users to turn the logging back on if they need it.
      *Result*: No endpoint requests are logged, including the `/health` endpoint.
      Show
      *Cause*: The monitoring-plugin swapped to a different backend which logged each request to it. An additional health check was added to the monitoring-plugin deployment, which causes the `/health` endpoint to get pinged every so often. *Consequence*: There is an excess of logs to the `/health` endpoint. *Fix*: Remove logging of requests as a default behavior. Add in an additional `log-verbosity` parameter to allow power users to turn the logging back on if they need it. *Result*: No endpoint requests are logged, including the `/health` endpoint.
    • Release Note Not Required
    • In Progress

      Description of problem:

          Multiple monitoring-plugin Pods return the response code every 10s, there will be too many logs as time goes by

      Version-Release number of selected component (if applicable):

          4.18.0-0.nightly-2024-09-09-150616

      How reproducible:

      Steps to Reproduce:

          1.
    2.
    3.

      Actual results:

      % oc -n openshift-monitoring logs monitoring-plugin-76b8c847f6-m872m
time="2024-09-10T07:55:52Z" level=info msg="enabled features: []\n" module=main
time="2024-09-10T07:55:52Z" level=warning msg="cannot read config file, serving plugin with default configuration, tried /etc/plugin/config.yaml" error="open /etc/plugin/config.yaml: no such file or directory" module=server
time="2024-09-10T07:55:52Z" level=info msg="listening on https://:9443" module=server
10.128.2.2 - - [10/Sep/2024:07:55:53 +0000] "GET /health HTTP/2.0" 200 2
10.128.2.2 - - [10/Sep/2024:07:55:58 +0000] "GET /health HTTP/2.0" 200 2
10.128.2.2 - - [10/Sep/2024:07:56:08 +0000] "GET /health HTTP/2.0" 200 2
10.128.2.2 - - [10/Sep/2024:07:56:18 +0000] "GET /health HTTP/2.0" 200 2
10.128.2.2 - - [10/Sep/2024:07:56:28 +0000] "GET /health HTTP/2.0" 200 2
10.128.2.2 - - [10/Sep/2024:07:56:38 +0000] "GET /health HTTP/2.0" 200 2
...

$ oc -n openshift-monitoring logs monitoring-plugin-76b8c847f6-m872m | grep "GET /health HTTP/2.0" | wc -l
1967

       

      Expected results:

          Before we switched to the golang backend, there are usually not many logs

      Additional info:

              rh-ee-pyurkovi Peter Yurkovich
              tagao@redhat.com Tai Gao
              Tai Gao Tai Gao
              Junqi Zhao
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated: