-
Bug
-
Resolution: Duplicate
-
Normal
-
Logging 5.7.0
-
False
-
None
-
False
-
NEW
-
NEW
-
Bug Fix
-
-
-
Important
Description of problem:
Deployed clusterlogging operator and fluentd server successfully on OCP 4.13 cluster. Created clusterlogging and clusterlogforward instance . But it fails to forward the logs to the fluentd server.
fluentd server resources
[root@rdr-ocp-mani-413-bastion-0 aosqe-tools-master]# oc get all NAME READY STATUS RESTARTS AGE pod/fluentdserver-fdb479f97-b46w8 1/1 Running 0 6h56mNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/fluentdserver ClusterIP 172.30.101.193 <none> 24224/TCP 8hNAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/fluentdserver 1/1 1 1 8hNAME DESIRED CURRENT READY AGE replicaset.apps/fluentdserver-fdb479f97 1 1 1 8h
No logs are stored in fluentdserver.
[root@rdr-ocp-mani-413-bastion-0 aosqe-tools-master]# oc rsh fluentdserver-fdb479f97-b46w8 ~ $ cd fluentd/log/ /fluentd/log $ ls /fluentd/log $ ls
CSV And Pods
[root@rdr-ocp-mani-413-bastion-0 ~]# oc get csv -n openshift-logging NAME DISPLAY VERSION REPLACES PHASE cluster-logging.v5.7.0 Red Hat OpenShift Logging 5.7.0 Succeeded elasticsearch-operator.v5.7.0 OpenShift Elasticsearch Operator 5.7.0 Succeeded loki-operator.v5.7.0 Loki Operator 5.7.0 Succeeded [root@rdr-ocp-mani-413-bastion-0 ~]# oc get pods -n openshift-logging NAME READY STATUS RESTARTS AGE cluster-logging-operator-784dfd6846-cgcfr 1/1 Running 1 (7h9m ago) 7h18m collector-bms2k 2/2 Running 0 47m collector-bxhdp 2/2 Running 0 47m collector-hn48w 2/2 Running 0 47m collector-r5cr7 2/2 Running 0 47m collector-w69vg 2/2 Running 0 47m lokistack-sample-compactor-0 1/1 Running 0 7h20m lokistack-sample-distributor-8559f9ddcb-nj9qj 1/1 Running 0 7h21m lokistack-sample-distributor-8559f9ddcb-xnhhn 1/1 Running 0 7h21m lokistack-sample-gateway-dcf4699f8-w86wj 2/2 Running 0 7h21m lokistack-sample-gateway-dcf4699f8-x7db5 2/2 Running 0 7h21m lokistack-sample-index-gateway-0 1/1 Running 0 7h20m lokistack-sample-index-gateway-1 1/1 Running 0 7h19m lokistack-sample-ingester-0 1/1 Running 0 7h20m lokistack-sample-ingester-1 1/1 Running 0 7h22m lokistack-sample-querier-846b977547-64dfv 1/1 Running 0 7h23m lokistack-sample-querier-846b977547-tl4fk 1/1 Running 0 7h21m lokistack-sample-query-frontend-77dd6f49b5-5c6cd 1/1 Running 0 7h21m lokistack-sample-query-frontend-77dd6f49b5-tqgss 1/1 Running 0 7h21m lokistack-sample-ruler-0 1/1 Running 0 7h22m lokistack-sample-ruler-1 1/1 Running 0 7h20m
**
Logs in the fluentd collector pods:
2023-05-09 16:56:41 +0000 [warn]: failed to flush the buffer. retry_times=0 next_retry_time=2023-05-09 16:56:42 +0000 chunk="5fb3ce885fa95fe12de75d6a085a5af6" error_class=Fluent::Plugin::HTTPOutput::RetryableResponse error="503 Service Unavailable <!DOCTYPE html PUBLIC \"-//W3C//DTD HTML 4.01//EN\" \"http://www.w3.org/TR/html4/strict.dtd\">\n<html><head>\n<meta type=\"copyright\" content=\"Copyright (C) 1996-2021 The Squid Software Foundation and contributors\">\n<meta http-equiv=\"Content-Type\" content=\"text/html; charset=utf-8\">\n<title>ERROR: The requested URL could not be retrieved</title>\n<style type=\"text/css\"><!-- \n /*\n * Copyright (C) 1996-2021 The Squid Software Foundation and contributors\n *\n * Squid software is distributed under GPLv2+ license and includes\n * contributions from numerous individuals and organizations.\n * Please see the COPYING and CONTRIBUTORS files for details.\n */\n\n/*\n Stylesheet for Squid Error pages\n Adapted from design by Free CSS Templates\n http://www.freecsstemplates.org\n Released for free under a Creative Commons Attribution 2.5 License\n*/\n\n/* Page basics */\n* {\n\tfont-family: verdana, sans-serif;\n}\n\nhtml body {\n\tmargin: 0;\n\tpadding: 0;\n\tbackground: #efefef;\n\tfont-size: 12px;\n\tcolor: #1e1e1e;\n}\n\n/* Page displayed title area */\n#titles {\n\tmargin-left: 15px;\n\tpadding: 10px;\n\tpadding-left: 100px;\n\tbackground: url('/squid-internal-static/icons/SN.png') no-repeat left;\n}\n\n/* initial title */\n#titles h1 {\n\tcolor: #000000;\n}\n#titles h2 {\n\tcolor: #000000;\n}\n\n/* special event: FTP success page titles */\n#titles ftpsuccess {\n\tbackground-color:#00ff00;\n\twidth:100%;\n}\n\n/* Page displayed body content area */\n#content {\n\tpadding: 10px;\n\tbackground: #ffffff;\n}\n\n/* General text */\np {\n}\n\n/* error brief description */\n#error p {\n}\n\n/* some data which may have caused the problem */\n#data {\n}\n\n/* the error message received from the system or other software */\n#sysmsg {\n}\n\npre {\n}\n\n/* special event: FTP / Gopher directory listing */\n#dirmsg {\n font-family: courier, monospace;\n color: black;\n font-size: 10pt;\n}\n#dirlisting {\n margin-left: 2%;\n margin-right: 2%;\n}\n#dirlisting tr.entry td.icon,td.filename,td.size,td.date {\n border-bottom: groove;\n}\n#dirlisting td.size {\n width: 50px;\n text-align: right;\n padding-right: 5px;\n}\n\n/* horizontal lines */\nhr {\n\tmargin: 0;\n}\n\n/* page displayed footer area */\n#footer {\n\tfont-size: 9px;\n\tpadding-left: 10px;\n}\n\n\nbody\n:lang(fa) { direction: rtl; font-size: 100%; font-family: Tahoma, Roya, sans-serif; float: right; }\n:lang(he) { direction: rtl; }\n --></style>\n</head><body id=ERR_DNS_FAIL>\n<div id=\"titles\">\n<h1>ERROR</h1>\n<h2>The requested URL could not be retrieved</h2>\n</div>\n<hr>\n\n<div id=\"content\">\n<p>The following error was encountered while trying to retrieve the URL: <a href=\"http://fluentdserver.fluentdserver.svc:24224/logs/audit\">http://fluentdserver.fluentdserver.svc:24224/logs/audit</a></p>\n\n<blockquote id=\"error\">\n<p><b>Unable to determine IP address from host name <q>fluentdserver.fluentdserver.svc</q></b></p>\n</blockquote>\n\n<p>The DNS server returned:</p>\n<blockquote id=\"data\">\n<pre>Name Error: The domain name does not exist.</pre>\n</blockquote>\n\n<p>This means that the cache was not able to resolve the hostname presented in the URL. Check if the address is correct.</p>\n\n<p>Your cache administrator is <a href=\"mailto:root?subject=CacheErrorInfo%20-%20ERR_DNS_FAIL&body=CacheHost%3A%20rdr-ocp-mani-413-bastion-0.rdr-ocp-mani-413.ibm.com%0D%0AErrPage%3A%20ERR_DNS_FAIL%0D%0AErr%3A%20%5Bnone%5D%0D%0ADNS%20ErrMsg%3A%20Name%20Error%3A%20The%20domain%20name%20does%20not%20exist.%0D%0ATimeStamp%3A%20Tue,%2009%20May%202023%2016%3A56%3A41%20GMT%0D%0A%0D%0AClientIP%3A%209.114.97.177%0D%0A%0D%0AHTTP%20Request%3A%0D%0APOST%20%2Flogs%2Faudit%20HTTP%2F1.1%0AAccept-Encoding%3A%20gzip%3Bq%3D1.0,deflate%3Bq%3D0.6,identity%3Bq%3D0.3%0D%0AAccept%3A%20*%2F*%0D%0AUser-Agent%3A%20Ruby%0D%0AH1%3A%20v1%0D%0AH2%3A%20v2%0D%0AContent-Type%3A%20application%2Fx-ndjson%0D%0AContent-Length%3A%208071782%0D%0AHost%3A%20fluentdserver.fluentdserver.svc%3A24224%0D%0A%0D%0A%0D%0A\">root</a>.</p>\n<br>\n</div>\n\n<hr>\n<div id=\"footer\">\n<p>Generated Tue, 09 May 2023 16:56:41 GMT by rdr-ocp-mani-413-bastion-0.rdr-ocp-mani-413.ibm.com (squid/4.15)</p>\n<!-- ERR_DNS_FAIL -->\n</div>\n</body></html>\n" 2023-05-09 16:56:41 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/out_http.rb:264:in `send_request' 2023-05-09 16:56:41 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/out_http.rb:148:in `write' 2023-05-09 16:56:41 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1179:in `try_flush' 2023-05-09 16:56:41 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1500:in `flush_thread_run' 2023-05-09 16:56:41 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:499:in `block (2 levels) in start' 2023-05-09 16:56:41 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create' 2023-05-09 16:56:42 +0000 [warn]: failed to flush the buffer. retry_times=0 next_retry_time=2023-05-09 16:56:43 +0000 chunk="5fb3ce89005733bb48b4f383e8015a35" error_class=Errno::ECONNRESET error="Connection reset by peer"
Version-Release number of selected component (if applicable):
Logging 5.7.0
How reproducible:
Always on IBM Power Platform for 5.7.0
Steps to Reproduce:
- Deploy any OCP version cluster 4.13 on the IBM Power.
- Install logging operators 5.7.0 on the OCP cluster.
- Deploy fluentd server on OCP cluster.
- Create Clusterlogging instance and create Clusterlogforwarder CR to forward logs to fluentdserver
- Check collelctor, clo pod logs
- Check the data in fluentdserver pod
Actual results:
Seeing error message in collector pod logs.
[root@rdr-ocp-mani-413-bastion-0 aosqe-tools-master]# oc logs collector-hn48w -n openshift-logging 2023-05-09 16:56:41 +0000 [warn]: failed to flush the buffer. retry_times=0 next_retry_time=2023-05-09 16:56:42 +0000 chunk="5fb3ce885fa95fe12de75d6a085a5af6" error_class=Fluent::Plugin::HTTPOutput::RetryableResponse error="503 Service Unavailable <!DOCTYPE html PUBLIC \"-//W3C//DTD HTML 4.01//EN\" \"http://www.w3.org/TR/html4/strict.dtd\">\n<html><head>\n<meta type=\"copyright\" content=\"Copyright (C) 1996-2021 The Squid Software Foundation and contributors\">\n<meta http-equiv=\"Content-Type\" content=\"text/html; charset=utf-8\">\n<title>ERROR: The requested URL could not be retrieved</title>\n<style type=\"text/css\"><!-- \n /*\n * Copyright (C) 1996-2021 The Squid Software Foundation and contributors\n *\n * Squid software is distributed under GPLv2+ license and includes\n * contributions from numerous individuals and organizations.\n * Please see the COPYING and CONTRIBUTORS files for details.\n */\n\n/*\n Stylesheet for Squid Error pages\n Adapted from design by Free CSS Templates\n http://www.freecsstemplates.org\n Released for free under a Creative Commons Attribution 2.5 License\n*/\n\n/* Page basics */\n* {\n\tfont-family: verdana, sans-serif;\n}\n\nhtml body {\n\tmargin: 0;\n\tpadding: 0;\n\tbackground: #efefef;\n\tfont-size: 12px;\n\tcolor: #1e1e1e;\n}\n\n/* Page displayed title area */\n#titles {\n\tmargin-left: 15px;\n\tpadding: 10px;\n\tpadding-left: 100px;\n\tbackground: url('/squid-internal-static/icons/SN.png') no-repeat left;\n}\n\n/* initial title */\n#titles h1 {\n\tcolor: #000000;\n}\n#titles h2 {\n\tcolor: #000000;\n}\n\n/* special event: FTP success page titles */\n#titles ftpsuccess {\n\tbackground-color:#00ff00;\n\twidth:100%;\n}\n\n/* Page displayed body content area */\n#content {\n\tpadding: 10px;\n\tbackground: #ffffff;\n}\n\n/* General text */\np {\n}\n\n/* error brief description */\n#error p {\n}\n\n/* some data which may have caused the problem */\n#data {\n}\n\n/* the error message received from the system or other software */\n#sysmsg {\n}\n\npre {\n}\n\n/* special event: FTP / Gopher directory listing */\n#dirmsg {\n font-family: courier, monospace;\n color: black;\n font-size: 10pt;\n}\n#dirlisting {\n margin-left: 2%;\n margin-right: 2%;\n}\n#dirlisting tr.entry td.icon,td.filename,td.size,td.date {\n border-bottom: groove;\n}\n#dirlisting td.size {\n width: 50px;\n text-align: right;\n padding-right: 5px;\n}\n\n/* horizontal lines */\nhr {\n\tmargin: 0;\n}\n\n/* page displayed footer area */\n#footer {\n\tfont-size: 9px;\n\tpadding-left: 10px;\n}\n\n\nbody\n:lang(fa) { direction: rtl; font-size: 100%; font-family: Tahoma, Roya, sans-serif; float: right; }\n:lang(he) { direction: rtl; }\n --></style>\n</head><body id=ERR_DNS_FAIL>\n<div id=\"titles\">\n<h1>ERROR</h1>\n<h2>The requested URL could not be retrieved</h2>\n</div>\n<hr>\n\n<div id=\"content\">\n<p>The following error was encountered while trying to retrieve the URL: <a href=\"http://fluentdserver.fluentdserver.svc:24224/logs/audit\">http://fluentdserver.fluentdserver.svc:24224/logs/audit</a></p>\n\n<blockquote id=\"error\">\n<p><b>Unable to determine IP address from host name <q>fluentdserver.fluentdserver.svc</q></b></p>\n</blockquote>\n\n<p>The DNS server returned:</p>\n<blockquote id=\"data\">\n<pre>Name Error: The domain name does not exist.</pre>\n</blockquote>\n\n<p>This means that the cache was not able to resolve the hostname presented in the URL. Check if the address is correct.</p>\n\n<p>Your cache administrator is <a href=\"mailto:root?subject=CacheErrorInfo%20-%20ERR_DNS_FAIL&body=CacheHost%3A%20rdr-ocp-mani-413-bastion-0.rdr-ocp-mani-413.ibm.com%0D%0AErrPage%3A%20ERR_DNS_FAIL%0D%0AErr%3A%20%5Bnone%5D%0D%0ADNS%20ErrMsg%3A%20Name%20Error%3A%20The%20domain%20name%20does%20not%20exist.%0D%0ATimeStamp%3A%20Tue,%2009%20May%202023%2016%3A56%3A41%20GMT%0D%0A%0D%0AClientIP%3A%209.114.97.177%0D%0A%0D%0AHTTP%20Request%3A%0D%0APOST%20%2Flogs%2Faudit%20HTTP%2F1.1%0AAccept-Encoding%3A%20gzip%3Bq%3D1.0,deflate%3Bq%3D0.6,identity%3Bq%3D0.3%0D%0AAccept%3A%20*%2F*%0D%0AUser-Agent%3A%20Ruby%0D%0AH1%3A%20v1%0D%0AH2%3A%20v2%0D%0AContent-Type%3A%20application%2Fx-ndjson%0D%0AContent-Length%3A%208071782%0D%0AHost%3A%20fluentdserver.fluentdserver.svc%3A24224%0D%0A%0D%0A%0D%0A\">root</a>.</p>\n<br>\n</div>\n\n<hr>\n<div id=\"footer\">\n<p>Generated Tue, 09 May 2023 16:56:41 GMT by rdr-ocp-mani-413-bastion-0.rdr-ocp-mani-413.ibm.com (squid/4.15)</p>\n<!-- ERR_DNS_FAIL -->\n</div>\n</body></html>\n" 2023-05-09 16:56:41 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/out_http.rb:264:in `send_request' 2023-05-09 16:56:41 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/out_http.rb:148:in `write' 2023-05-09 16:56:41 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1179:in `try_flush' 2023-05-09 16:56:41 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1500:in `flush_thread_run' 2023-05-09 16:56:41 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:499:in `block (2 levels) in start' 2023-05-09 16:56:41 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create' 2023-05-09 16:56:42 +0000 [warn]: failed to flush the buffer. retry_times=0 next_retry_time=2023-05-09 16:56:43 +0000 chunk="5fb3ce89005733bb48b4f383e8015a35" error_class=Errno::ECONNRESET error="Connection reset by peer"
Expected results:
forward logs to fluentd over http succesfully.