-
Bug
-
Resolution: Won't Do
-
Normal
-
None
-
Logging 5.6.7
-
False
-
None
-
False
-
NEW
-
NEW
-
Bug Fix
-
-
-
Important
Description of problem:
It's visible in fluentd the stacktrace:
$ for pod in $(oc get pods -l component=collector -o name -n openshift-logging); do echo -e "### POD: $pod ###" ;oc logs $pod -c collector -n openshift-logging|grep -c "unwatched=' for nil:NilClass"; done ### POD: pod/collector-2j6mn ### 0 ### POD: pod/collector-4gnxt ### 0 ### POD: pod/collector-58vtk ### 0 ### POD: pod/collector-8n9x6 ### 0 ### POD: pod/collector-cbltt ### 0 ### POD: pod/collector-g22tq ### 0 ### POD: pod/collector-gh7d4 ### 0 ### POD: pod/collector-jtbkr ### 998 ### POD: pod/collector-kpf47 ### 0 ### POD: pod/collector-q5ccv ### 23 ### POD: pod/collector-z5crz ### 0 ### POD: pod/collector-zfb5d ### 703 $ oc logs pod/collector-jtbkr -n openshift-logging -c collector ... 2023-09-20T09:39:15.140186354Z 2023-09-20 09:39:15 +0000 [error]: [container-input] undefined method `unwatched=' for nil:NilClass 2023-09-20T09:39:15.140186354Z 2023-09-20 09:39:15 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:534:in `update_watcher' 2023-09-20T09:39:15.140186354Z 2023-09-20 09:39:15 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:902:in `call' 2023-09-20T09:39:15.140186354Z 2023-09-20 09:39:15 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:902:in `on_rotate' 2023-09-20T09:39:15.140186354Z 2023-09-20 09:39:15 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:1201:in `on_notify' 2023-09-20T09:39:15.140186354Z 2023-09-20 09:39:15 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:835:in `on_notify' 2023-09-20T09:39:15.140186354Z 2023-09-20 09:39:15 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:403:in `block in setup_watcher' 2023-09-20T09:39:15.140186354Z 2023-09-20 09:39:15 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:755:in `on_change' 2023-09-20T09:39:15.140186354Z 2023-09-20 09:39:15 +0000 [error]: /usr/local/share/gems/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run_once' 2023-09-20T09:39:15.140186354Z 2023-09-20 09:39:15 +0000 [error]: /usr/local/share/gems/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run' 2023-09-20T09:39:15.140186354Z 2023-09-20 09:39:15 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start' 2023-09-20T09:39:15.140186354Z 2023-09-20 09:39:15 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Another error is visible in the collectors indicating that not able to `expand_paths`:
$ for pod in $(oc get pods -l component=collector -o name -n openshift-logging); do echo -e "### POD: $pod ###" ;oc logs $pod -c collector -n openshift-logging|grep -c expand_paths; done ### POD: pod/collector-2j6mn ### 0 ### POD: pod/collector-4gnxt ### 0 ### POD: pod/collector-58vtk ### 1 ### POD: pod/collector-8n9x6 ### 0 ### POD: pod/collector-cbltt ### 0 ### POD: pod/collector-g22tq ### 0 ### POD: pod/collector-gh7d4 ### 0 ### POD: pod/collector-jtbkr ### 0 ### POD: pod/collector-kpf47 ### 0 ### POD: pod/collector-q5ccv ### 0 ### POD: pod/collector-z5crz ### 0 ### POD: pod/collector-zfb5d ### 1
Also, it's visible these pods are hitting LOG-4241 and from one moment in the time for the future, not able to ingest more logs from some pods (the same reported in https://issues.redhat.com/browse/LOG-3633 and LOG-4241)
This same stacktrace was reported in https://issues.redhat.com/browse/OCPBUGS-13246, but closed referencing to LOG-3633.
Needed to indicated that fix for LOG-3633 for released with Logging 5.6.6 and the current bug is being reported for 5.6.7.
Version-Release number of selected component (if applicable):
CLO 5.6.7
How reproducible:
Not able to reproduce, only visible the stacktrace on the collector pods
Actual results:
Stacktrace is observed:
Bug for error 2023-09-20T09:39:15.140186354Z 2023-09-20 09:39:15 +0000 [error]: [container-input] undefined method unwatched=' for nil:NilClass will be opened
Expected results:
In OCPBUGS-13246 is indicated this stacktrace could be only be noise, but an stacktrace per se indicates an exception not controlled. Then, as it's only missing logs from one moment to the future and also, not able to release properly the pod logs deleted LOG-4241, it's expected to review the stacktrace and from where it's coming and also, manage that exception.
