Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-3762

[fluentd over http ]encoding "application/x-ndjson" is an invalid parameter

    XMLWordPrintable

Details

    • False
    • None
    • False
    • NEW
    • NEW
    • Hide
      Before this update, the Cluster Logging generate an invalid parameter for flunetd http plugin 'encoding "application/x-ndjson"', but it should be 'content_type: "application/x-ndjson"'. With this update, Cluster Logging generate a correct configuration for fluentd http plugin.

      Show
      Before this update, the Cluster Logging generate an invalid parameter for flunetd http plugin 'encoding "application/x-ndjson"', but it should be 'content_type: "application/x-ndjson"'. With this update, Cluster Logging generate a correct configuration for fluentd http plugin.
    • Log Collection - Sprint 233, Log Collection - Sprint 234

    Description

      The clo generate an invalid parameter 'encoding "application/x-ndjson"'. It should be content_type: "application/x-ndjson". refer to https://docs.fluentd.org/output/http

      Step to reproduce:

      1. Deploy CLO.
      2. Create fluentd receiver.

      3. Forward logs to fluentd receiver over http:
      {
          "apiVersion": "logging.openshift.io/v1",
          "kind": "ClusterLogForwarder",
          "metadata": {
              "name": "instance",
              "namespace": "openshift-logging"
          },
          "spec": {
              "outputs": [
                  {
                      "http": {
                          "headers": {
                              "h1": "v1",
                              "h2": "v2"
                          },
                          "method": "POST"
                      },
                      "name": "httpout-app",
                      "type": "http",
                      "url": "http://fluentdserver-fluentdserver.apps.anli13.qe.devcluster.openshift.com/logs/app"
                  },
                  {
                      "http": {
                          "headers": {
                              "h1": "v1",
                              "h2": "v2"
                          },
                          "method": "POST"
                      },
                      "name": "httpout-infra",
                      "type": "http",
                      "url": "http://fluentdserver-fluentdserver.apps.anli13.qe.devcluster.openshift.com/logs/infra"
                  },
                  {
                      "http": {
                          "headers": {
                              "h1": "v1",
                              "h2": "v2"
                          },
                          "method": "POST"
                      },
                      "name": "httpout-audit",
                      "type": "http",
                      "url": "http://fluentdserver-fluentdserver.apps.anli13.qe.devcluster.openshift.com/logs/audit"
                  }
              ],
              "pipelines": [
                  {
                      "inputRefs": [
                          "application"
                      ],
                      "name": "app-logs",
                      "outputRefs": [
                          "httpout-app"
                      ]
                  },
                  {
                      "inputRefs": [
                          "infrastructure"
                      ],
                      "name": "infra-logs",
                      "outputRefs": [
                          "httpout-infra"
                      ]
                  },
                  {
                      "inputRefs": [
                          "audit"
                      ],
                      "name": "audit-logs",
                      "outputRefs": [
                          "httpout-audit"
                      ]
                  }
              ]
          }
      }
      

      4. Check the fluentd pod and fluent.conf

      Actual result:

      2023-03-13 08:37:42 +0000 [warn]: parameter 'encoding' in <match **>
      @type http
      endpoint "http://fluentdserver-fluentdserver.apps.anli13.qe.devcluster.openshift.com/logs/audit"
      http_method post
      encoding application/x-ndjson
      headers

      {"h1":"v1","h2":"v2"}

      <buffer>
      @type "file"
      path "/var/lib/fluentd/httpout_audit"
      flush_mode interval
      flush_interval 1s
      flush_thread_count 2
      retry_type exponential_backoff
      retry_wait 1s
      retry_max_interval 60s
      retry_timeout 60m
      queued_chunks_limit_size 573
      total_limit_size 4809148876
      chunk_limit_size 8388608
      overflow_action block
      disable_chunk_backup true
      </buffer>
      </match> is not used.

      <label @HTTPOUT_AUDIT>
        <match **>
          @type http
          endpoint http://fluentdserver-fluentdserver.apps.anli13.qe.devcluster.openshift.com/logs/audit
          http_method post
          encoding "application/x-ndjson"
          headers {"h1":"v1","h2":"v2"}
      
          <buffer>
            @type file
            path '/var/lib/fluentd/httpout_audit'
            flush_mode interval
            flush_interval 1s
            flush_thread_count 2
            retry_type exponential_backoff
            retry_wait 1s
            retry_max_interval 60s
            retry_timeout 60m
            queued_chunks_limit_size "#{ENV['BUFFER_QUEUE_LIMIT'] || '32'}"
            total_limit_size "#{ENV['TOTAL_LIMIT_SIZE_PER_BUFFER'] || '8589934592'}"
            chunk_limit_size "#{ENV['BUFFER_SIZE_LIMIT'] || '8m'}"
            overflow_action block
            disable_chunk_backup true
          </buffer>
        </match>
      

      Attachments

        Activity

          People

            vparfono Vitalii Parfonov
            rhn-support-anli Anping Li
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: