Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-3559

After logging operator upgraded to 5.6.0, collectors restart every 5 minutes.

    XMLWordPrintable

Details

    • False
    • None
    • False
    • NEW
    • VERIFIED
    • Hide
      Before this update, when the `ClusterLogForwarder` custom resource (CR) had multiple pipelines configured, with one output set as `default`, the collector pods restarted. With this update, the logic for output validation has been corrected, resolving the issue.
      Show
      Before this update, when the `ClusterLogForwarder` custom resource (CR) had multiple pipelines configured, with one output set as `default`, the collector pods restarted. With this update, the logic for output validation has been corrected, resolving the issue.
    • Log Collection - Sprint 231, Log Collection - Sprint 232

    Description

      Description of problem:

      Collector pods get deleted after a couple of minutes and recreated. Setting CL instance to Unmanaged, the restarts stop.

      Version-Release number of selected component (if applicable):

      5.6.0

      How reproducible:

      Occurs in multiple clusters on customer side, not successfully reproduced in lab

      Steps to Reproduce:

      1. Upgraded from 5.5 to 5.6
      2. Set CL instance to Managed...

      Actual results:

      Events show:
      Tue Jan 24 09:31:00 CST 2023 openshift-logging Warning Invalid clusterlogforwarder/instance invalid: unrecognized outputs: [default], no valid outputs
      Mon Jan 23 04:47:57 CST 2023 openshift-logging Warning Invalid clusterlogforwarder/instance

      CLO logs only repeat:

      {{}}

      2023-01-19T17:28:03.202102402Z {"_ts":"2023-01-19T17:28:03.20110673Z","_level":"0","_component":"cluster-logging-operator","_message":"clusterlogging-controller error updating status","_error":{"msg":"Operation cannot be fulfilled on clusterloggings.logging.openshift.io \"instance\": the object has been modified; please apply your changes to the latest version and try again"}} 
      

      CLF:

       
      spec:
        outputs:
        - name: rsyslog-prod
          syslog:
            appName: example
            facility: user
            msgID: mymsg
            procID: myproc
            rfc: RFC5424
            severity: informational
          type: syslog
          url: udp://syslog.example.com:514
        pipelines:
        - inputRefs:
          - audit
          labels:
            syslog: example-prod
          name: syslog-prod
          outputRefs:
          - rsyslog-prod
          parse: json
        - inputRefs:
          - application
          - infrastructure
          name: enable-default-logs
          outputRefs:
          - default
      

       

      When the CLO is set to Unmanaged, all logs flow normally to syslog and elasticsearch, so the configs themselves seem to work just fine.

      Expected results:

      I dont see any errors in the config; I would expect the CLO to either NOT restart pods, or else to log what it thinks the error is.

      Additional info:

      Attachments

        Activity

          People

            cahartma@redhat.com Casey Hartman
            rhn-support-stwalter Steven Walter
            Qiaoling Tang Qiaoling Tang
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: