Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-3784

[fluentd http] the defaut value HTTP content type application/x-ndjson is unsupported on datadog

    XMLWordPrintable

Details

    • False
    • None
    • False
    • Engineering
    • NEW
    • NEW
    • Hide
      Before this changes, CLO got exception "Unsupported Media Type" then try to send logs to DataDog via Fluentd HTTP Plugin. With this update, users can assign the content type for log forwarding seamlessly, by configuring the HTTP header "Content-Type," the value provided is automatically assigned to the content_type parameter within the plugin, as result logs will send successfully.
      Show
      Before this changes, CLO got exception "Unsupported Media Type" then try to send logs to DataDog via Fluentd HTTP Plugin. With this update, users can assign the content type for log forwarding seamlessly, by configuring the HTTP header "Content-Type," the value provided is automatically assigned to the content_type parameter within the plugin, as result logs will send successfully.
    • Bug Fix
    • Log Collection - Sprint 234, Log Collection - Sprint 235, Log Collection - Sprint 236

    Description

      Content_type "application/x-ndjson" is generated by default. this parameter causes two issues.
      1) The default value is not supported by datadog
      2) The default value can not be overwritten by header

      <label @DATADOG_ANLI>
      <match **>
      @type http
      endpoint https://http-intake.logs.us5.datadoghq.com/api/v2/logs
      http_method post
      content_type "application/x-ndjson"
      headers

      {"DD-API-KEY":"xxxxx"}

      .....
      </match>
      </label>

      Step to Reproduce:

      1. Forward logs to datadog over http

      cat <<EOF|oc create -f -
      apiVersion: "logging.openshift.io/v1"
      kind: "ClusterLogging"
      metadata:
        name: "instance"
        namespace: openshift-logging
      spec:
        managementState: "Managed"
        collection:
          type: "fluentd"
      
      apiVersion: logging.openshift.io/v1
      kind: ClusterLogForwarder
      metadata:
        name: instance
        namespace: openshift-logging
      spec:
        outputs:
        - http:
            headers:
              DD-API-KEY: xxxxxx
              Content-Type: application/json
            method: POST
          name: datadog-anli
          type: http
          url: https://http-intake.logs.us5.datadoghq.com/api/v2/logs
        pipelines:
        - inputRefs:
          - application
          name:  to-datadog
          outputRefs:
          - datadog-anli
      EOF
      

      2. Check the collect pod logs

      Actual result:

      The datadog return message 'error="415 Unsupported Media Type'

      2023-03-15 10:15:50 +0000 [warn]: suppressed same stacktrace
      2023-03-15 10:15:50 +0000 [warn]: disable_chunk_backup is true. 5f6eda0d82a8828c1c2aa964cba76dda chunk is thrown away
      2023-03-15 10:15:51 +0000 [warn]: got unrecoverable error in primary and no secondary error_class=Fluent::UnrecoverableError error="415 Unsupported Media Type {\"errors\":[

      {\"status\":\"415\",\"title\":\"Unsupported Media Type\",\"detail\":\"HTTP content type is unsupported\"}

      ]}"

      Expected result:

      CLO don't set "application/x-ndjson" as the default Content-Type.

      <label @DATADOG_ANLI>
      <match **>
      @type http
      endpoint https://http-intake.logs.us5.datadoghq.com/api/v2/logs
      http_method post
      content_type "application/x-ndjson"
      headers

      {"Content-Type":"application/json","DD-API-KEY":"xxx"}

      .....
      </match>

      Attachments

        Activity

          People

            vparfono Vitalii Parfonov
            anli@redhat.com Anping Li
            Anping Li Anping Li
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: