Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-4053

[ppc64le] Forward logs to fluentd over http - http failed (cluster logging v5.7)

XMLWordPrintable

    • False
    • None
    • False
    • NEW
    • NEW
    • Bug Fix
    • Important

      Description of problem:

      Deployed clusterlogging operator and  fluentd server successfully on OCP 4.13 cluster. Created clusterlogging  and clusterlogforward instance . But it fails to forward the logs to the fluentd server.

       

      fluentd server resources

      [root@rdr-ocp-mani-413-bastion-0 aosqe-tools-master]# oc get all
NAME                                READY   STATUS    RESTARTS   AGE
pod/fluentdserver-fdb479f97-b46w8   1/1     Running   0          6h56mNAME                    TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)     AGE
service/fluentdserver   ClusterIP   172.30.101.193   <none>        24224/TCP   8hNAME                            READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/fluentdserver   1/1     1            1           8hNAME                                      DESIRED   CURRENT   READY   AGE
replicaset.apps/fluentdserver-fdb479f97   1         1         1       8h

       

      No logs are stored in fluentdserver.

      [root@rdr-ocp-mani-413-bastion-0 aosqe-tools-master]# oc rsh fluentdserver-fdb479f97-b46w8
~ $ cd fluentd/log/
/fluentd/log $ ls
/fluentd/log $ ls

      CSV And Pods

      [root@rdr-ocp-mani-413-bastion-0 ~]# oc get csv -n openshift-logging
NAME                            DISPLAY                            VERSION   REPLACES   PHASE
cluster-logging.v5.7.0          Red Hat OpenShift Logging          5.7.0                Succeeded
elasticsearch-operator.v5.7.0   OpenShift Elasticsearch Operator   5.7.0                Succeeded
loki-operator.v5.7.0            Loki Operator                      5.7.0                Succeeded
[root@rdr-ocp-mani-413-bastion-0 ~]# oc get pods -n openshift-logging
NAME                                               READY   STATUS    RESTARTS       AGE
cluster-logging-operator-784dfd6846-cgcfr          1/1     Running   1 (7h9m ago)   7h18m
collector-bms2k                                    2/2     Running   0              47m
collector-bxhdp                                    2/2     Running   0              47m
collector-hn48w                                    2/2     Running   0              47m
collector-r5cr7                                    2/2     Running   0              47m
collector-w69vg                                    2/2     Running   0              47m
lokistack-sample-compactor-0                       1/1     Running   0              7h20m
lokistack-sample-distributor-8559f9ddcb-nj9qj      1/1     Running   0              7h21m
lokistack-sample-distributor-8559f9ddcb-xnhhn      1/1     Running   0              7h21m
lokistack-sample-gateway-dcf4699f8-w86wj           2/2     Running   0              7h21m
lokistack-sample-gateway-dcf4699f8-x7db5           2/2     Running   0              7h21m
lokistack-sample-index-gateway-0                   1/1     Running   0              7h20m
lokistack-sample-index-gateway-1                   1/1     Running   0              7h19m
lokistack-sample-ingester-0                        1/1     Running   0              7h20m
lokistack-sample-ingester-1                        1/1     Running   0              7h22m
lokistack-sample-querier-846b977547-64dfv          1/1     Running   0              7h23m
lokistack-sample-querier-846b977547-tl4fk          1/1     Running   0              7h21m
lokistack-sample-query-frontend-77dd6f49b5-5c6cd   1/1     Running   0              7h21m
lokistack-sample-query-frontend-77dd6f49b5-tqgss   1/1     Running   0              7h21m
lokistack-sample-ruler-0                           1/1     Running   0              7h22m
lokistack-sample-ruler-1                           1/1     Running   0              7h20m

                                  **                            

      Logs in the fluentd collector pods:

      2023-05-09 16:56:41 +0000 [warn]: failed to flush the buffer. retry_times=0 next_retry_time=2023-05-09 16:56:42 +0000 chunk="5fb3ce885fa95fe12de75d6a085a5af6" error_class=Fluent::Plugin::HTTPOutput::RetryableResponse error="503 Service Unavailable <!DOCTYPE html PUBLIC \"-//W3C//DTD HTML 4.01//EN\" \"http://www.w3.org/TR/html4/strict.dtd\">\n<html><head>\n<meta type=\"copyright\" content=\"Copyright (C) 1996-2021 The Squid Software Foundation and contributors\">\n<meta http-equiv=\"Content-Type\" content=\"text/html; charset=utf-8\">\n<title>ERROR: The requested URL could not be retrieved</title>\n<style type=\"text/css\"><!-- \n /*\n * Copyright (C) 1996-2021 The Squid Software Foundation and contributors\n *\n * Squid software is distributed under GPLv2+ license and includes\n * contributions from numerous individuals and organizations.\n * Please see the COPYING and CONTRIBUTORS files for details.\n */\n\n/*\n Stylesheet for Squid Error pages\n Adapted from design by Free CSS Templates\n http://www.freecsstemplates.org\n Released for free under a Creative Commons Attribution 2.5 License\n*/\n\n/* Page basics */\n* {\n\tfont-family: verdana, sans-serif;\n}\n\nhtml body {\n\tmargin: 0;\n\tpadding: 0;\n\tbackground: #efefef;\n\tfont-size: 12px;\n\tcolor: #1e1e1e;\n}\n\n/* Page displayed title area */\n#titles {\n\tmargin-left: 15px;\n\tpadding: 10px;\n\tpadding-left: 100px;\n\tbackground: url('/squid-internal-static/icons/SN.png') no-repeat left;\n}\n\n/* initial title */\n#titles h1 {\n\tcolor: #000000;\n}\n#titles h2 {\n\tcolor: #000000;\n}\n\n/* special event: FTP success page titles */\n#titles ftpsuccess {\n\tbackground-color:#00ff00;\n\twidth:100%;\n}\n\n/* Page displayed body content area */\n#content {\n\tpadding: 10px;\n\tbackground: #ffffff;\n}\n\n/* General text */\np {\n}\n\n/* error brief description */\n#error p {\n}\n\n/* some data which may have caused the problem */\n#data {\n}\n\n/* the error message received from the system or other software */\n#sysmsg {\n}\n\npre {\n}\n\n/* special event: FTP / Gopher directory listing */\n#dirmsg {\n    font-family: courier, monospace;\n    color: black;\n    font-size: 10pt;\n}\n#dirlisting {\n    margin-left: 2%;\n    margin-right: 2%;\n}\n#dirlisting tr.entry td.icon,td.filename,td.size,td.date {\n    border-bottom: groove;\n}\n#dirlisting td.size {\n    width: 50px;\n    text-align: right;\n    padding-right: 5px;\n}\n\n/* horizontal lines */\nhr {\n\tmargin: 0;\n}\n\n/* page displayed footer area */\n#footer {\n\tfont-size: 9px;\n\tpadding-left: 10px;\n}\n\n\nbody\n:lang(fa) { direction: rtl; font-size: 100%; font-family: Tahoma, Roya, sans-serif; float: right; }\n:lang(he) { direction: rtl; }\n --></style>\n</head><body id=ERR_DNS_FAIL>\n<div id=\"titles\">\n<h1>ERROR</h1>\n<h2>The requested URL could not be retrieved</h2>\n</div>\n<hr>\n\n<div id=\"content\">\n<p>The following error was encountered while trying to retrieve the URL: <a href=\"http://fluentdserver.fluentdserver.svc:24224/logs/audit\">http://fluentdserver.fluentdserver.svc:24224/logs/audit</a></p>\n\n<blockquote id=\"error\">\n<p><b>Unable to determine IP address from host name <q>fluentdserver.fluentdserver.svc</q></b></p>\n</blockquote>\n\n<p>The DNS server returned:</p>\n<blockquote id=\"data\">\n<pre>Name Error: The domain name does not exist.</pre>\n</blockquote>\n\n<p>This means that the cache was not able to resolve the hostname presented in the URL. Check if the address is correct.</p>\n\n<p>Your cache administrator is <a href=\"mailto:root?subject=CacheErrorInfo%20-%20ERR_DNS_FAIL&amp;body=CacheHost%3A%20rdr-ocp-mani-413-bastion-0.rdr-ocp-mani-413.ibm.com%0D%0AErrPage%3A%20ERR_DNS_FAIL%0D%0AErr%3A%20%5Bnone%5D%0D%0ADNS%20ErrMsg%3A%20Name%20Error%3A%20The%20domain%20name%20does%20not%20exist.%0D%0ATimeStamp%3A%20Tue,%2009%20May%202023%2016%3A56%3A41%20GMT%0D%0A%0D%0AClientIP%3A%209.114.97.177%0D%0A%0D%0AHTTP%20Request%3A%0D%0APOST%20%2Flogs%2Faudit%20HTTP%2F1.1%0AAccept-Encoding%3A%20gzip%3Bq%3D1.0,deflate%3Bq%3D0.6,identity%3Bq%3D0.3%0D%0AAccept%3A%20*%2F*%0D%0AUser-Agent%3A%20Ruby%0D%0AH1%3A%20v1%0D%0AH2%3A%20v2%0D%0AContent-Type%3A%20application%2Fx-ndjson%0D%0AContent-Length%3A%208071782%0D%0AHost%3A%20fluentdserver.fluentdserver.svc%3A24224%0D%0A%0D%0A%0D%0A\">root</a>.</p>\n<br>\n</div>\n\n<hr>\n<div id=\"footer\">\n<p>Generated Tue, 09 May 2023 16:56:41 GMT by rdr-ocp-mani-413-bastion-0.rdr-ocp-mani-413.ibm.com (squid/4.15)</p>\n<!-- ERR_DNS_FAIL -->\n</div>\n</body></html>\n"
  2023-05-09 16:56:41 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/out_http.rb:264:in `send_request'
  2023-05-09 16:56:41 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/out_http.rb:148:in `write'
  2023-05-09 16:56:41 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1179:in `try_flush'
  2023-05-09 16:56:41 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1500:in `flush_thread_run'
  2023-05-09 16:56:41 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:499:in `block (2 levels) in start'
  2023-05-09 16:56:41 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2023-05-09 16:56:42 +0000 [warn]: failed to flush the buffer. retry_times=0 next_retry_time=2023-05-09 16:56:43 +0000 chunk="5fb3ce89005733bb48b4f383e8015a35" error_class=Errno::ECONNRESET error="Connection reset by peer"

      Version-Release number of selected component (if applicable):

      Logging 5.7.0

      How reproducible:

      Always on IBM Power Platform for 5.7.0

      Steps to Reproduce:

      1. Deploy any OCP version cluster 4.13 on the IBM Power.
      2. Install logging operators 5.7.0 on the OCP cluster.
      3. Deploy fluentd server on OCP cluster.
      4. Create Clusterlogging instance and create Clusterlogforwarder CR to forward logs to fluentdserver
      5. Check collelctor, clo pod logs
      6. Check the data in fluentdserver pod

      Actual results:

      Seeing error message in collector pod logs.

      [root@rdr-ocp-mani-413-bastion-0 aosqe-tools-master]# oc logs collector-hn48w -n openshift-logging
2023-05-09 16:56:41 +0000 [warn]: failed to flush the buffer. retry_times=0 next_retry_time=2023-05-09 16:56:42 +0000 chunk="5fb3ce885fa95fe12de75d6a085a5af6" error_class=Fluent::Plugin::HTTPOutput::RetryableResponse error="503 Service Unavailable <!DOCTYPE html PUBLIC \"-//W3C//DTD HTML 4.01//EN\" \"http://www.w3.org/TR/html4/strict.dtd\">\n<html><head>\n<meta type=\"copyright\" content=\"Copyright (C) 1996-2021 The Squid Software Foundation and contributors\">\n<meta http-equiv=\"Content-Type\" content=\"text/html; charset=utf-8\">\n<title>ERROR: The requested URL could not be retrieved</title>\n<style type=\"text/css\"><!-- \n /*\n * Copyright (C) 1996-2021 The Squid Software Foundation and contributors\n *\n * Squid software is distributed under GPLv2+ license and includes\n * contributions from numerous individuals and organizations.\n * Please see the COPYING and CONTRIBUTORS files for details.\n */\n\n/*\n Stylesheet for Squid Error pages\n Adapted from design by Free CSS Templates\n http://www.freecsstemplates.org\n Released for free under a Creative Commons Attribution 2.5 License\n*/\n\n/* Page basics */\n* {\n\tfont-family: verdana, sans-serif;\n}\n\nhtml body {\n\tmargin: 0;\n\tpadding: 0;\n\tbackground: #efefef;\n\tfont-size: 12px;\n\tcolor: #1e1e1e;\n}\n\n/* Page displayed title area */\n#titles {\n\tmargin-left: 15px;\n\tpadding: 10px;\n\tpadding-left: 100px;\n\tbackground: url('/squid-internal-static/icons/SN.png') no-repeat left;\n}\n\n/* initial title */\n#titles h1 {\n\tcolor: #000000;\n}\n#titles h2 {\n\tcolor: #000000;\n}\n\n/* special event: FTP success page titles */\n#titles ftpsuccess {\n\tbackground-color:#00ff00;\n\twidth:100%;\n}\n\n/* Page displayed body content area */\n#content {\n\tpadding: 10px;\n\tbackground: #ffffff;\n}\n\n/* General text */\np {\n}\n\n/* error brief description */\n#error p {\n}\n\n/* some data which may have caused the problem */\n#data {\n}\n\n/* the error message received from the system or other software */\n#sysmsg {\n}\n\npre {\n}\n\n/* special event: FTP / Gopher directory listing */\n#dirmsg {\n    font-family: courier, monospace;\n    color: black;\n    font-size: 10pt;\n}\n#dirlisting {\n    margin-left: 2%;\n    margin-right: 2%;\n}\n#dirlisting tr.entry td.icon,td.filename,td.size,td.date {\n    border-bottom: groove;\n}\n#dirlisting td.size {\n    width: 50px;\n    text-align: right;\n    padding-right: 5px;\n}\n\n/* horizontal lines */\nhr {\n\tmargin: 0;\n}\n\n/* page displayed footer area */\n#footer {\n\tfont-size: 9px;\n\tpadding-left: 10px;\n}\n\n\nbody\n:lang(fa) { direction: rtl; font-size: 100%; font-family: Tahoma, Roya, sans-serif; float: right; }\n:lang(he) { direction: rtl; }\n --></style>\n</head><body id=ERR_DNS_FAIL>\n<div id=\"titles\">\n<h1>ERROR</h1>\n<h2>The requested URL could not be retrieved</h2>\n</div>\n<hr>\n\n<div id=\"content\">\n<p>The following error was encountered while trying to retrieve the URL: <a href=\"http://fluentdserver.fluentdserver.svc:24224/logs/audit\">http://fluentdserver.fluentdserver.svc:24224/logs/audit</a></p>\n\n<blockquote id=\"error\">\n<p><b>Unable to determine IP address from host name <q>fluentdserver.fluentdserver.svc</q></b></p>\n</blockquote>\n\n<p>The DNS server returned:</p>\n<blockquote id=\"data\">\n<pre>Name Error: The domain name does not exist.</pre>\n</blockquote>\n\n<p>This means that the cache was not able to resolve the hostname presented in the URL. Check if the address is correct.</p>\n\n<p>Your cache administrator is <a href=\"mailto:root?subject=CacheErrorInfo%20-%20ERR_DNS_FAIL&amp;body=CacheHost%3A%20rdr-ocp-mani-413-bastion-0.rdr-ocp-mani-413.ibm.com%0D%0AErrPage%3A%20ERR_DNS_FAIL%0D%0AErr%3A%20%5Bnone%5D%0D%0ADNS%20ErrMsg%3A%20Name%20Error%3A%20The%20domain%20name%20does%20not%20exist.%0D%0ATimeStamp%3A%20Tue,%2009%20May%202023%2016%3A56%3A41%20GMT%0D%0A%0D%0AClientIP%3A%209.114.97.177%0D%0A%0D%0AHTTP%20Request%3A%0D%0APOST%20%2Flogs%2Faudit%20HTTP%2F1.1%0AAccept-Encoding%3A%20gzip%3Bq%3D1.0,deflate%3Bq%3D0.6,identity%3Bq%3D0.3%0D%0AAccept%3A%20*%2F*%0D%0AUser-Agent%3A%20Ruby%0D%0AH1%3A%20v1%0D%0AH2%3A%20v2%0D%0AContent-Type%3A%20application%2Fx-ndjson%0D%0AContent-Length%3A%208071782%0D%0AHost%3A%20fluentdserver.fluentdserver.svc%3A24224%0D%0A%0D%0A%0D%0A\">root</a>.</p>\n<br>\n</div>\n\n<hr>\n<div id=\"footer\">\n<p>Generated Tue, 09 May 2023 16:56:41 GMT by rdr-ocp-mani-413-bastion-0.rdr-ocp-mani-413.ibm.com (squid/4.15)</p>\n<!-- ERR_DNS_FAIL -->\n</div>\n</body></html>\n"
  2023-05-09 16:56:41 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/out_http.rb:264:in `send_request'
  2023-05-09 16:56:41 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/out_http.rb:148:in `write'
  2023-05-09 16:56:41 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1179:in `try_flush'
  2023-05-09 16:56:41 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1500:in `flush_thread_run'
  2023-05-09 16:56:41 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:499:in `block (2 levels) in start'
  2023-05-09 16:56:41 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2023-05-09 16:56:42 +0000 [warn]: failed to flush the buffer. retry_times=0 next_retry_time=2023-05-09 16:56:43 +0000 chunk="5fb3ce89005733bb48b4f383e8015a35" error_class=Errno::ECONNRESET error="Connection reset by peer"

      Expected results:

      forward logs to fluentd over http succesfully.

      Additional info:

        1. CollectorDemonset.txt
          21 kB

              Unassigned Unassigned
              rhn-ee-magaikwa Manisha Gaikwad
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: