Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-3482

Lots of noisy message in cluster-logging-operator pod logs

    XMLWordPrintable

Details

    • False
    • None
    • False
    • NEW
    • NEW
    • Hide
      Without this update Cluster Logging Operator pod logs contained numerous noisy messages, leading to reduced log readability and increased difficulty in identifying important system events.
      With this update, significant reduction in noisy messages within Cluster Logging Operator pod logs, resulting in improved log readability and enhanced visibility of critical system events.
      Show
      Without this update Cluster Logging Operator pod logs contained numerous noisy messages, leading to reduced log readability and increased difficulty in identifying important system events. With this update, significant reduction in noisy messages within Cluster Logging Operator pod logs, resulting in improved log readability and enhanced visibility of critical system events.
    • Log Collection - Sprint 233, Log Collection - Sprint 234, Log Collection - Sprint 235, Log Collection - Sprint 236

    Description

      Descriptions:
      logs of noisy messages in cluster-logging-operator pod logs. we can group these message into three types. these message should be warning rather than error.

      1. No valid inputs found
      2. the object has been modified; please apply your changes to the latest version and try again
      3. "configmaps "grafana-dashboard-cluster-logging" not found"

      oc logs cluster-logging-operator-8cff696f7-mdk5w
      ....
      cluster-logging-operator","_message":"Error reconciling clusterlogging instance","_error":{"msg":"unable to create or update collection for \"instance\": No valid inputs found in ClusterLogForwarder"}}
      {"_ts":"2023-01-10T11:10:16.932344082Z","_level":"0","_component":"cluster-logging-operator","_message":"Error retrieving the Trusted CA Bundle","_error":{"msg":"ConfigMap \"collector-trusted-ca-bundle\" not found"}}
      {"_ts":"2023-01-10T11:10:22.133140943Z","_level":"0","_component":"cluster-logging-operator","_message":"clusterlogging-controller error updating status","_error":{"msg":"Operation cannot be fulfilled on clusterloggings.logging.openshift.io \"instance\": the object has been modified; please apply your changes to the latest version and try again"}}
      {"_ts":"2023-01-10T11:15:04.691448257Z","_level":"0","_component":"cluster-logging-operator","_message":"error deleting grafana configmap","_error":{"msg":"configmaps \"grafana-dashboard-cluster-logging\" not found"}}

      Steps to reproduce:
      1) deploy clusterlogging/intance.
      2) check the cluster-logging-operator pod logs

      Actual Result:
      Lots of noisy messages

      {"_ts":"2023-01-10T11:08:59.155485656Z","_level":"0","_component":"cluster-logging-operator","_message":"starting up...","go_arch":"amd64","go_os":"linux","go_version":"go1.17.12","operator_version":"5.6"}

      I0110 11:09:00.206515 1 request.go:665] Waited for 1.02804805s due to client-side throttling, not priority and fairness, request: GET:https://172.30.0.1:443/apis/ingress.operator.openshift.io/v1?timeout=32s

      {"_ts":"2023-01-10T11:09:01.519622369Z","_level":"0","_component":"cluster-logging-operator","_message":"migrating resources provided by the manifest"} {"_ts":"2023-01-10T11:09:01.525408612Z","_level":"0","_component":"cluster-logging-operator","_message":"Registering Components."} {"_ts":"2023-01-10T11:09:01.525615719Z","_level":"0","_component":"cluster-logging-operator","_message":"Starting the Cmd."}

      {"_ts":"2023-01-10T11:09:55.357017327Z","_level":"0","_component":"cluster-logging-operator","_message":"Unable to generate log configuration","_error":{"msg":"No valid inputs found in ClusterLogForwarder"}}
      {"_ts":"2023-01-10T11:09:55.371794049Z","_level":"0","_component":"cluster-logging-operator","_message":"clusterRequest.generateCollectorConfig","_error":{"msg":"No valid inputs found in ClusterLogForwarder"}}
      {"_ts":"2023-01-10T11:09:55.37189608Z","_level":"0","_component":"cluster-logging-operator","_message":"Error reconciling clusterlogging instance","_error":{"msg":"unable to create or update collection for \"instance\": No valid inputs found in ClusterLogForwarder"}}
      {"_ts":"2023-01-10T11:09:55.380569492Z","_level":"0","_component":"cluster-logging-operator","_message":"clusterlogging-controller error updating status","_error":{"msg":"Operation cannot be fulfilled on clusterloggings.logging.openshift.io \"instance\": the object has been modified; please apply your changes to the latest version and try again"}}
      {"_ts":"2023-01-10T11:09:55.461163617Z","_level":"0","_component":"cluster-logging-operator","_message":"Unable to generate log configuration","_error":{"msg":"No valid inputs found in ClusterLogForwarder"}}
      {"_ts":"2023-01-10T11:09:55.461332896Z","_level":"0","_component":"cluster-logging-operator","_message":"clusterRequest.generateCollectorConfig","_error":{"msg":"No valid inputs found in ClusterLogForwarder"}}
      {"_ts":"2023-01-10T11:09:55.461398201Z","_level":"0","_component":"cluster-logging-operator","_message":"Error reconciling clusterlogging instance","_error":{"msg":"unable to create or update collection for \"instance\": No valid inputs found in ClusterLogForwarder"}}
      {"_ts":"2023-01-10T11:09:55.471257576Z","_level":"0","_component":"cluster-logging-operator","_message":"clusterlogging-controller error updating status","_error":{"msg":"Operation cannot be fulfilled on clusterloggings.logging.openshift.io \"instance\": the object has been modified; please apply your changes to the latest version and try again"}}
      {"_ts":"2023-01-10T11:09:55.540032214Z","_level":"0","_component":"cluster-logging-operator","_message":"Unable to generate log configuration","_error":{"msg":"No valid inputs found in ClusterLogForwarder"}}
      {"_ts":"2023-01-10T11:09:55.540135359Z","_level":"0","_component":"cluster-logging-operator","_message":"clusterRequest.generateCollectorConfig","_error":{"msg":"No valid inputs found in ClusterLogForwarder"}}
      {"_ts":"2023-01-10T11:09:55.540183127Z","_level":"0","_component":"cluster-logging-operator","_message":"Error reconciling clusterlogging instance","_error":{"msg":"unable to create or update collection for \"instance\": No valid inputs found in ClusterLogForwarder"}}
      {...................................

      {"_ts":"2023-01-10T11:10:22.133140943Z","_level":"0","_component":"cluster-logging-operator","_message":"clusterlogging-controller error updating status","_error":{"msg":"Operation cannot be fulfilled on clusterloggings.logging.openshift.io \"instance\": the object has been modified; please apply your changes to the latest version and try again"}}
      {"_ts":"2023-01-10T11:10:22.546694459Z","_level":"0","_component":"cluster-logging-operator","_message":"clusterlogging-controller error updating status","_error":{"msg":"Operation cannot be fulfilled on clusterloggings.logging.openshift.io \"instance\": the object has been modified; please apply your changes to the latest version and try again"}}
      {"_ts":"2023-01-10T11:10:22.801555831Z","_level":"0","_component":"cluster-logging-operator","_message":"clusterlogging-controller error updating status","_error":{"msg":"Operation cannot be fulfilled on clusterloggings.logging.openshift.io \"instance\": the object has been modified; please apply your changes to the latest version and try again"}}
      {"_ts":"2023-01-10T11:10:23.018580467Z","_level":"0","_component":"cluster-logging-operator","_message":"clusterlogging-controller error updating status","_error":{"msg":"Operation cannot be fulfilled on clusterloggings.logging.openshift.io \"instance\": the object has been modified; please apply your changes to the latest version and try again"}}
      {"_ts":"2023-01-10T11:10:23.332166292Z","_level":"0","_component":"cluster-logging-operator","_message":"clusterlogging-controller error updating status","_error":{"msg":"Operation cannot be fulfilled on clusterloggings.logging.openshift.io \"instance\": the object has been modified; please apply your changes to the latest version and try again"}}
      {"_ts":"2023-01-10T11:10:25.701550823Z","_level":"0","_component":"cluster-logging-operator","_message":"Error updating status","_error":{"msg":"Operation cannot be fulfilled on clusterloggings.logging.openshift.io \"instance\": the object has been modified; please apply your changes to the latest version and try again"}}
      {"_ts":"2023-01-10T11:10:25.890814092Z","_level":"0","_component":"cluster-logging-operator","_message":"clusterlogging-controller error updating status","_error":{"msg":"Operation cannot be fulfilled on clusterloggings.logging.openshift.io \"instance\": the object has been modified; please apply your changes to the latest version and try again"}}
      {"_ts":"2023-01-10T11:10:26.075720348Z","_level":"0","_component":"cluster-logging-operator","_message":"clusterlogging-controller error updating status","_error":{"msg":"Operation cannot be fulfilled on clusterloggings.logging.openshift.io \"instance\": the object has been modified; please apply your changes to the latest version and try again"}}
      {"_ts":"2023-01-10T11:10:26.284976567Z","_level":"0","_component":"cluster-logging-operator","_message":"clusterlogging-controller error updating status","_error":{"msg":"Operation cannot be fulfilled on clusterloggings.logging.openshift.io \"instance\": the object has been modified; please apply your changes to the latest version and try again"}}
      {"_ts":"2023-01-10T11:10:27.320321019Z","_level":"0","_component":"cluster-logging-operator","_message":"clusterlogging-controller error updating status","_error":{"msg":"Operation cannot be fulfilled on clusterloggings.logging.openshift.io \"instance\": the object has been modified; please apply your changes to the latest version and try again"}}
      {"_ts":"2023-01-10T11:14:36.484610401Z","_level":"0","_component":"cluster-logging-operator","_message":"Error updating status","_error":{"msg":"Operation cannot be fulfilled on clusterloggings.logging.openshift.io \"instance\": the object has been modified; please apply your changes to the latest version and try again"}}
      {"_ts":"2023-01-10T11:14:36.598542135Z","_level":"0","_component":"cluster-logging-operator","_message":"clusterlogging-controller error updating status","_error":{"msg":"Operation cannot be fulfilled on clusterloggings.logging.openshift.io \"instance\": the object has been modified; please apply your changes to the latest version and try again"}}
      {"_ts":"2023-01-10T11:15:04.567226611Z","_level":"0","_component":"cluster-logging-operator","_message":"error deleting grafana configmap","_error":{"msg":"configmaps \"grafana-dashboard-cluster-logging\" not found"}}
      {"_ts":"2023-01-10T11:15:04.647359107Z","_level":"0","_component":"cluster-logging-operator","_message":"error deleting grafana configmap","_error":{"msg":"configmaps \"grafana-dashboard-cluster-logging\" not found"}}
      {"_ts":"2023-01-10T11:15:04.680185379Z","_level":"0","_component":"cluster-logging-operator","_message":"error deleting grafana configmap","_error":{"msg":"configmaps \"grafana-dashboard-cluster-logging\" not found"}}
      {"_ts":"2023-01-10T11:15:04.691448257Z","_level":"0","_component":"cluster-logging-operator","_message":"error deleting grafana configmap","_error":{"msg":"configmaps \"grafana-dashboard-cluster-logging\" not found"}}
      {"_ts":"2023-01-10T11:15:04.895713025Z","_level":"0","_component":"cluster-logging-operator","_message":"error deleting grafana configmap","_error":{"msg":"configmaps \"grafana-dashboard-cluster-logging\" not found"}}
      {"_ts":"2023-01-10T11:20:33.117588622Z","_level":"0","_component":"cluster-logging-operator","_message":"clusterlogforwarder-controller returning, error","_error":{"msg":"ClusterLogging.logging.openshift.io \"instance\" not found"}}
      {"_ts":"2023-01-10T11:20:33.12448292Z","_level":"0","_component":"cluster-logging-operator","_message":"error deleting grafana configmap","_error":{"msg":"configmaps \"grafana-dashboard-cluster-logging\" not found"}}
      {"_ts":"2023-01-10T11:20:33.131833128Z","_level":"0","_component":"cluster-logging-operator","_message":"clusterlogforwarder-controller returning, error","_error":{"msg":"ClusterLogging.logging.openshift.io \"instance\" not found"}}
      {"_ts":"2023-01-10T11:20:33.137404885Z","_level":"0","_component":"cluster-logging-operator","_message":"error deleting grafana configmap","_error":{"msg":"configmaps \"grafana-dashboard-cluster-logging\" not found"}}
      {

      Attachments

        Activity

          People

            vparfono Vitalii Parfonov
            rhn-support-anli Anping Li
            Ishwar Kanse Ishwar Kanse
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: