Loading...

Type: Bug
Resolution: Done
Priority: Major
Fix Version/s: Logging 5.2
Affects Version/s: Logging 5.2
Component/s: Log Collection
Labels:

Blocked:
False
Ready:
False
Docs QE Status:
NEW
QE Status:
VERIFIED
Release Note Text:

Hide
Before this update, enabling the new fluentd `log_collected_bytes_total` metric by using the new `https://github.com/openshift/origin-aggregated-logging/tree/master/fluentd/lib/fluent-plugin-collected` plugin had issues: It applied the `namespace`, `podname`, and `containername` labels to this metric even when parsing the log file path names could not provide values for those labels. This condition threw the `Prometheus::Client::LabelSetValidator::InvalidLabelSetError` error.
This update fixes the issue: if parsing the log file path cannot produce values for those labels, their values are set to `notfound`.

Show
Before this update, enabling the new fluentd `log_collected_bytes_total` metric by using the new ` https://github.com/openshift/origin-aggregated-logging/tree/master/fluentd/lib/fluent-plugin-collected ` plugin had issues: It applied the `namespace`, `podname`, and `containername` labels to this metric even when parsing the log file path names could not provide values for those labels. This condition threw the `Prometheus::Client::LabelSetValidator::InvalidLabelSetError` error. This update fixes the issue: if parsing the log file path cannot produce values for those labels, their values are set to `notfound`.
Market:

Sprint:
Logging (Core) - Sprint 205, Logging (Core) - Sprint 206

SFDC Cases Links:
SFDC Cases Counter:
SFDC Cases Open:

Description of problem:

Fluentd pod always raise below errors when forward logs to external ES/fluentd/rsyslog server:

Setting each total_size_limit for 3 buffers to 6415029657 bytes
Setting queued_chunks_limit_size for each buffer to 764
Setting chunk_limit_size for each buffer to 8388608
2021-08-05 06:40:54 +0000 [warn]: '@' is the system reserved prefix. It works in the nested configuration for now but it will be rejected: @timestamp
2021-08-05 06:40:55 +0000 [warn]: /var/log/ovn/acl-audit-log.log not found. Continuing without tailing it.
2021-08-05 06:41:31 +0000 [error]: Unexpected error raised. Stopping the timer. title=:in_collected_tail_monitor error_class=Prometheus::Client::LabelSetValidator::InvalidLabelSetError error="labels must have the same signature (keys given: [:hostname, :path, :plugin_id, :type] vs. keys expected: [:containername, :hostname, :namespace, :path, :plugin_id, :podname, :type]"
  2021-08-05 06:41:31 +0000 [error]: /usr/local/share/gems/gems/prometheus-client-0.9.0/lib/prometheus/client/label_set_validator.rb:38:in `validate'
  2021-08-05 06:41:31 +0000 [error]: /usr/local/share/gems/gems/prometheus-client-0.9.0/lib/prometheus/client/metric.rb:65:in `label_set_for'
  2021-08-05 06:41:31 +0000 [error]: /usr/local/share/gems/gems/prometheus-client-0.9.0/lib/prometheus/client/counter.rb:16:in `increment'
  2021-08-05 06:41:31 +0000 [error]: /usr/local/share/gems/gems/fluent-plugin-collected-1.0.0/lib/fluent/plugin/in_collected_tail_monitor.rb:102:in `block (2 levels) in update_monitor_info'
  2021-08-05 06:41:31 +0000 [error]: /usr/local/share/gems/gems/fluent-plugin-collected-1.0.0/lib/fluent/plugin/in_collected_tail_monitor.rb:84:in `each'
  2021-08-05 06:41:31 +0000 [error]: /usr/local/share/gems/gems/fluent-plugin-collected-1.0.0/lib/fluent/plugin/in_collected_tail_monitor.rb:84:in `block in update_monitor_info'
  2021-08-05 06:41:31 +0000 [error]: /usr/local/share/gems/gems/fluent-plugin-collected-1.0.0/lib/fluent/plugin/in_collected_tail_monitor.rb:80:in `each'
  2021-08-05 06:41:31 +0000 [error]: /usr/local/share/gems/gems/fluent-plugin-collected-1.0.0/lib/fluent/plugin/in_collected_tail_monitor.rb:80:in `update_monitor_info'
  2021-08-05 06:41:31 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin_helper/timer.rb:80:in `on_timer'
  2021-08-05 06:41:31 +0000 [error]: /usr/local/share/gems/gems/cool.io-1.6.0/lib/cool.io/loop.rb:88:in `run_once'
  2021-08-05 06:41:31 +0000 [error]: /usr/local/share/gems/gems/cool.io-1.6.0/lib/cool.io/loop.rb:88:in `run'
  2021-08-05 06:41:31 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
  2021-08-05 06:41:31 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2021-08-05 06:41:31 +0000 [error]: Timer detached. title=:in_collected_tail_monitor

Version-Release number of selected component (if applicable):

cluster-logging.5.2.0-21

How reproducible:

Always

Steps to Reproduce:

1. deploy an external ES/fluentd/rsyslog server
2. create clusterlogging/instance and only deploy fluentd pods
3. create clusterlogforwarder/instance, forward logs to external ES/fluentd/rsyslog server
4. check fluentd pod logs

Actual results:

Expected results:

no error in fluentd

Additional info: