Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-1647

Fluentd pods raise error `Prometheus::Client::LabelSetValidator::InvalidLabelSetError` when forward logs to external logStore

    XMLWordPrintable

Details

    • Logging (Core) - Sprint 205, Logging (Core) - Sprint 206
    • Passed
    • NEW
    • VERIFIED
    • Hide
      Before this update, enabling the new fluentd `log_collected_bytes_total` metric by using the new `https://github.com/openshift/origin-aggregated-logging/tree/master/fluentd/lib/fluent-plugin-collected` plugin had issues: It applied the `namespace`, `podname`, and `containername` labels to this metric even when parsing the log file path names could not provide values for those labels. This condition threw the `Prometheus::Client::LabelSetValidator::InvalidLabelSetError` error.
      This update fixes the issue: if parsing the log file path cannot produce values for those labels, their values are set to `notfound`.
      Show
      Before this update, enabling the new fluentd `log_collected_bytes_total` metric by using the new ` https://github.com/openshift/origin-aggregated-logging/tree/master/fluentd/lib/fluent-plugin-collected ` plugin had issues: It applied the `namespace`, `podname`, and `containername` labels to this metric even when parsing the log file path names could not provide values for those labels. This condition threw the `Prometheus::Client::LabelSetValidator::InvalidLabelSetError` error. This update fixes the issue: if parsing the log file path cannot produce values for those labels, their values are set to `notfound`.

    Description

      Description of problem:

      Fluentd pod always raise below errors when forward logs to external ES/fluentd/rsyslog server:

      Setting each total_size_limit for 3 buffers to 6415029657 bytes
      Setting queued_chunks_limit_size for each buffer to 764
      Setting chunk_limit_size for each buffer to 8388608
      2021-08-05 06:40:54 +0000 [warn]: '@' is the system reserved prefix. It works in the nested configuration for now but it will be rejected: @timestamp
      2021-08-05 06:40:55 +0000 [warn]: /var/log/ovn/acl-audit-log.log not found. Continuing without tailing it.
      2021-08-05 06:41:31 +0000 [error]: Unexpected error raised. Stopping the timer. title=:in_collected_tail_monitor error_class=Prometheus::Client::LabelSetValidator::InvalidLabelSetError error="labels must have the same signature (keys given: [:hostname, :path, :plugin_id, :type] vs. keys expected: [:containername, :hostname, :namespace, :path, :plugin_id, :podname, :type]"
        2021-08-05 06:41:31 +0000 [error]: /usr/local/share/gems/gems/prometheus-client-0.9.0/lib/prometheus/client/label_set_validator.rb:38:in `validate'
        2021-08-05 06:41:31 +0000 [error]: /usr/local/share/gems/gems/prometheus-client-0.9.0/lib/prometheus/client/metric.rb:65:in `label_set_for'
        2021-08-05 06:41:31 +0000 [error]: /usr/local/share/gems/gems/prometheus-client-0.9.0/lib/prometheus/client/counter.rb:16:in `increment'
        2021-08-05 06:41:31 +0000 [error]: /usr/local/share/gems/gems/fluent-plugin-collected-1.0.0/lib/fluent/plugin/in_collected_tail_monitor.rb:102:in `block (2 levels) in update_monitor_info'
        2021-08-05 06:41:31 +0000 [error]: /usr/local/share/gems/gems/fluent-plugin-collected-1.0.0/lib/fluent/plugin/in_collected_tail_monitor.rb:84:in `each'
        2021-08-05 06:41:31 +0000 [error]: /usr/local/share/gems/gems/fluent-plugin-collected-1.0.0/lib/fluent/plugin/in_collected_tail_monitor.rb:84:in `block in update_monitor_info'
        2021-08-05 06:41:31 +0000 [error]: /usr/local/share/gems/gems/fluent-plugin-collected-1.0.0/lib/fluent/plugin/in_collected_tail_monitor.rb:80:in `each'
        2021-08-05 06:41:31 +0000 [error]: /usr/local/share/gems/gems/fluent-plugin-collected-1.0.0/lib/fluent/plugin/in_collected_tail_monitor.rb:80:in `update_monitor_info'
        2021-08-05 06:41:31 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin_helper/timer.rb:80:in `on_timer'
        2021-08-05 06:41:31 +0000 [error]: /usr/local/share/gems/gems/cool.io-1.6.0/lib/cool.io/loop.rb:88:in `run_once'
        2021-08-05 06:41:31 +0000 [error]: /usr/local/share/gems/gems/cool.io-1.6.0/lib/cool.io/loop.rb:88:in `run'
        2021-08-05 06:41:31 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
        2021-08-05 06:41:31 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
      2021-08-05 06:41:31 +0000 [error]: Timer detached. title=:in_collected_tail_monitor

      Version-Release number of selected component (if applicable):

      cluster-logging.5.2.0-21

      How reproducible:

      Always

      Steps to Reproduce:

      1. deploy an external ES/fluentd/rsyslog server
      2. create clusterlogging/instance and only deploy fluentd pods
      3. create clusterlogforwarder/instance, forward logs to external ES/fluentd/rsyslog server
      4. check fluentd pod logs 

      Actual results:

      Expected results:

      no error in fluentd 

      Additional info: 

      Attachments

        Activity

          People

            pmoogi pratibha moogi
            qitang@redhat.com Qiaoling Tang
            Qiaoling Tang Qiaoling Tang
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: