Uploaded image for project: 'OpenShift Windows Containers'
  1. OpenShift Windows Containers
  2. WINC-1150

Change log level on inconsequential spammy messages

XMLWordPrintable

    • Icon: Story Story
    • Resolution: Done
    • Icon: Major Major
    • WMCO 10.16.0
    • None
    • None
    • 2
    • False
    • None
    • False
    • WINC - Sprint 253

      Description

      When debugging user WMCO logs the logs have an extremely high signal to noise ratio. For example the WMCO log:

      2023-10-17T03:01:43.879085023Z {"level":"info","ts":"2023-10-17T03:01:43Z","logger":"nc 10.168.1.74","msg":"instance has been configured as a worker node","version":"7.1.0-cfb3eef"}
      2023-10-17T03:01:43.966036792Z {"level":"info","ts":"2023-10-17T03:01:43Z","logger":"metrics","msg":"Prometheus configured","endpoints":"windows-exporter","port":9182,"name":"metrics"}
      2023-10-17T03:02:14.063112466Z {"level":"info","ts":"2023-10-17T03:02:14Z","logger":"metrics","msg":"Prometheus configured","endpoints":"windows-exporter","port":9182,"name":"metrics"}
      2023-10-17T03:07:50.336419472Z {"level":"info","ts":"2023-10-17T03:07:50Z","logger":"metrics","msg":"Prometheus configured","endpoints":"windows-exporter","port":9182,"name":"metrics"}
      2023-10-17T03:08:20.401926283Z {"level":"info","ts":"2023-10-17T03:08:20Z","logger":"metrics","msg":"Prometheus configured","endpoints":"windows-exporter","port":9182,"name":"metrics"}
      2023-10-17T03:43:02.424760112Z {"level":"info","ts":"2023-10-17T03:43:02Z","logger":"metrics","msg":"Prometheus configured","endpoints":"windows-exporter","port":9182,"name":"metrics"}
      2023-10-17T03:43:32.470413136Z {"level":"info","ts":"2023-10-17T03:43:32Z","logger":"metrics","msg":"Prometheus configured","endpoints":"windows-exporter","port":9182,"name":"metrics"}
      2023-10-17T03:51:01.635773798Z {"level":"info","ts":"2023-10-17T03:51:01Z","logger":"metrics","msg":"Prometheus configured","endpoints":"windows-exporter","port":9182,"name":"metrics"}
      2023-10-17T03:51:31.706251940Z {"level":"info","ts":"2023-10-17T03:51:31Z","logger":"metrics","msg":"Prometheus configured","endpoints":"windows-exporter","port":9182,"name":"metrics"}
      2023-10-17T03:53:05.062636459Z {"level":"info","ts":"2023-10-17T03:53:05Z","logger":"metrics","msg":"Prometheus configured","endpoints":"windows-exporter","port":9182,"name":"metrics"}
      2023-10-17T03:53:35.135526319Z {"level":"info","ts":"2023-10-17T03:53:35Z","logger":"metrics","msg":"Prometheus configured","endpoints":"windows-exporter","port":9182,"name":"metrics"}
      2023-10-17T04:27:25.113028967Z {"level":"info","ts":"2023-10-17T04:27:25Z","logger":"metrics","msg":"Prometheus configured","endpoints":"windows-exporter","port":9182,"name":"metrics"}
      2023-10-17T04:27:55.196227737Z {"level":"info","ts":"2023-10-17T04:27:55Z","logger":"metrics","msg":"Prometheus configured","endpoints":"windows-exporter","port":9182,"name":"metrics"}
      2023-10-17T05:54:52.546245655Z {"level":"info","ts":"2023-10-17T05:54:52Z","logger":"metrics","msg":"Prometheus configured","endpoints":"windows-exporter","port":9182,"name":"metrics"}
      2023-10-17T05:55:22.644175970Z {"level":"info","ts":"2023-10-17T05:55:22Z","logger":"metrics","msg":"Prometheus configured","endpoints":"windows-exporter","port":9182,"name":"metrics"}
      2023-10-17T06:09:01.121974563Z {"level":"info","ts":"2023-10-17T06:09:01Z","logger":"metrics","msg":"Prometheus configured","endpoints":"windows-exporter","port":9182,"name":"metrics"}
      2023-10-17T06:09:31.194252062Z {"level":"info","ts":"2023-10-17T06:09:31Z","logger":"metrics","msg":"Prometheus configured","endpoints":"windows-exporter","port":9182,"name":"metrics"}
      2023-10-17T07:57:48.173824489Z {"level":"info","ts":"2023-10-17T07:57:48Z","logger":"metrics","msg":"Prometheus configured","endpoints":"windows-exporter","port":9182,"name":"metrics"}
      2023-10-17T07:58:22.993766758Z {"level":"info","ts":"2023-10-17T07:58:22Z","logger":"metrics","msg":"Prometheus configured","endpoints":"windows-exporter","port":9182,"name":"metrics"}
      2023-10-17T08:01:14.325268071Z {"level":"info","ts":"2023-10-17T08:01:14Z","logger":"metrics","msg":"Prometheus configured","endpoints":"windows-exporter","port":9182,"name":"metrics"}
      2023-10-17T08:01:44.534034139Z {"level":"info","ts":"2023-10-17T08:01:44Z","logger":"metrics","msg":"Prometheus configured","endpoints":"windows-exporter","port":9182,"name":"metrics"}
      2023-10-17T08:02:14.628267636Z {"level":"info","ts":"2023-10-17T08:02:14Z","logger":"metrics","msg":"Prometheus configured","endpoints":"windows-exporter","port":9182,"name":"metrics"}
      2023-10-17T08:02:44.782324744Z {"level":"info","ts":"2023-10-17T08:02:44Z","logger":"metrics","msg":"Prometheus configured","endpoints":"windows-exporter","port":9182,"name":"metrics"}
      2023-10-17T08:08:10.405611220Z {"level":"info","ts":"2023-10-17T08:08:10Z","logger":"controllers.configmap","msg":"updating kubelet CA client certificates in","node":"winwor-z1-r8xvt"}
      

      and the WICD log, which is printing out these messages every 2 minutes:

      I1012 07:15:31.646933    5532 controller.go:290] successfully reconciled service kube-proxy
      I1012 07:15:31.664627    5532 controller.go:220] reconciling /winwor-z2-jbj24
      I1012 07:15:31.666234    5532 controller.go:285] reconciling existing service containerd
      I1012 07:15:32.457166    5532 controller.go:290] successfully reconciled service containerd
      I1012 07:15:32.457166    5532 controller.go:285] reconciling existing service kubelet
      I1012 07:15:32.457166    5532 controller.go:290] successfully reconciled service kubelet
      I1012 07:15:32.457672    5532 controller.go:285] reconciling existing service windows_exporter
      I1012 07:15:32.457699    5532 controller.go:290] successfully reconciled service windows_exporter
      I1012 07:15:32.457699    5532 controller.go:285] reconciling existing service hybrid-overlay-node
      I1012 07:15:32.457699    5532 controller.go:290] successfully reconciled service hybrid-overlay-node
      I1012 07:15:32.457699    5532 controller.go:285] reconciling existing service kube-proxy
      I1012 07:15:35.770141    5532 controller.go:290] successfully reconciled service kube-proxy
      I1012 07:16:53.666209    5532 controller.go:220] reconciling /winwor-z2-jbj24
      I1012 07:16:53.667274    5532 controller.go:285] reconciling existing service containerd
      I1012 07:16:54.898733    5532 controller.go:290] successfully reconciled service containerd
      I1012 07:16:54.898733    5532 controller.go:285] reconciling existing service kubelet
      I1012 07:16:54.898733    5532 controller.go:290] successfully reconciled service kubelet 

      Having these logs print when debugging logging is great, but they are quick to fill up the log files, and make the log files less valuable when trying to quickly determine if there is an issue occurring.

      Acceptance Criteria

      • The prometheus configured log message only logs when a real change to the metric endpoint has been made.
      • WICD no longer logs inconsequential messages when reconciling. Reconciliation logs that are not errors or calling out that a service was updated should be moved to the debug log level, on a case by case basis.

            rh-ee-ssoto Sebastian Soto
            rh-ee-ssoto Sebastian Soto
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: