-
Bug
-
Resolution: Won't Do
-
Major
-
Logging 5.4.z, Logging 5.5.z
-
False
-
None
-
False
-
NEW
-
NEW
-
-
-
Log Collection - Sprint 230, Log Collection - Sprint 231, Log Collection - Sprint 232, Log Collection - Sprint 233, Log Collection - Sprint 234, Log Collection - Sprint 236
Description of problem:
Collector pod intermittently not sending logs to external kafka
Version-Release number of selected component (if applicable):
$ oc get csv -n openshift-logging NAME DISPLAY VERSION REPLACES PHASE cluster-logging.5.5.4 Red Hat OpenShift Logging 5.5.4 cluster-logging.5.4.7 Succeeded elasticsearch-operator.5.5.4 OpenShift Elasticsearch Operator 5.5.4 elasticsearch-operator.5.5.3 Succeeded
How reproducible:
NA
Additional info:
logs of collector pod(collector-gknrg) : 2022-11-15 08:50:43 +0000 [warn]: [app_logs] chunk bytes limit exceeds for an emitted event stream: 3303866bytes 2022-11-15 08:50:43 +0000 [warn]: [app_logs] Send exception occurred: closed stream 2022-11-15 08:50:43 +0000 [warn]: [app_logs] Exception Backtrace : /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/buffer/file_chunk.rb:170:in `seek' /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/buffer/file_chunk.rb:170:in `open' /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/event.rb:314:in `each' /usr/local/share/gems/gems/fluent-plugin-kafka-0.17.5/lib/fluent/plugin/out_kafka2.rb:247:in `write' /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1179:in `try_flush' /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1500:in `flush_thread_run' /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:499:in `block (2 levels) in start' /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create' 2022-11-15 08:50:43 +0000 [warn]: [app_logs] failed to flush the buffer. retry_times=1 next_retry_time=2022-11-15 08:50:46 +0000 chunk="5ed792142cf607ecca448fd4a5fbc83d" error_class=IOError error="closed stream" 2022-11-15 08:50:43 +0000 [warn]: suppressed same stacktrace 2022-11-15 08:50:43 +0000 [warn]: [app_logs] Send exception occurred: closed stream 2022-11-15 08:50:43 +0000 [warn]: [app_logs] Exception Backtrace : /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/buffer/file_chunk.rb:170:in `seek' /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/buffer/file_chunk.rb:170:in `open' /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/event.rb:314:in `each' /usr/local/share/gems/gems/fluent-plugin-kafka-0.17.5/lib/fluent/plugin/out_kafka2.rb:247:in `write' /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1179:in `try_flush' /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1500:in `flush_thread_run' /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:499:in `block (2 levels) in start' /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create' 2022-11-15 08:50:43 +0000 [warn]: [app_logs] Send exception occurred: closed stream 2022-11-15 08:50:43 +0000 [warn]: [app_logs] Exception Backtrace : /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/buffer/file_chunk.rb:170:in `seek' /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/buffer/file_chunk.rb:170:in `open' /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/event.rb:314:in `each' /usr/local/share/gems/gems/fluent-plugin-kafka-0.17.5/lib/fluent/plugin/out_kafka2.rb:247:in `write' /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1179:in `try_flush' /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1500:in `flush_thread_run' /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:499:in `block (2 levels) in start' /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create' 2022-11-15 08:50:43 +0000 [warn]: [app_logs] failed to flush the buffer. retry_times=1 next_retry_time=2022-11-15 08:50:46 +0000 chunk="5ed7921d031571dddd5bcb7787acdb96" error_class=IOError error="closed stream"
$ oc rsh collector-gknrg Defaulted container "collector" out of: collector, logfilesmetricexporter sh-4.4# cd /var/lib/fluentd/ sh-4.4# du -sch * 27G app_logs ---> buffer full 0 audit_logs 12K infra_logs 880K pos 27G total
