Uploaded image for project: 'Observability and Data Analysis Program'
  1. Observability and Data Analysis Program
  2. OBSDA-1101

Transformation and restructing of fields in Log Forwarder

XMLWordPrintable

    • Icon: Feature Feature
    • Resolution: Unresolved
    • Icon: Normal Normal
    • None
    • Logging 6.1
    • PM Logging
    • None
    • False
    • None
    • False
    • Not Selected
    • 0
    • Important

      Proposed title of this feature request

      Transformation and restructing of fields in Log Forwarder

      What is the nature and description of the request?

      Log receivers at customers often have specific requirements for how logs should be received. This request is asking for a feature that allows a remapping / transform of fields in log messages. This would allow customers to define their own fields / schema for log entries and to customise the fields that are sent to the receiver.

      One specific use case would be when customers would like to use the Splunk HEC as a log receiver. The Splunk HEC endpoint has specific keys for `event` and `fields`, where an audit log would look like this:

      {
          "time": 1426279439, //convert requestReceivedTimestamp into epoch time
          "host": "example-master-2", //hostname
          "source": "http:openshift-example", //has to be a static value set by cluster-admin
          "sourcetype": "httpevent",  //has to be a static value set by cluster-admin
          "index": "example_index",  //has to be a static value set by cluster-admin
          "event":  "/api/v1/namespaces/testing/secrets?allowWatchBookmarks=true", //contains "RequestURI" for KubernetesAPI audit logs or "message" for linux audit logs
          "fields":  //contains the whole audit log
          {
              "annotations_authorization.k8s.io/decision": "allow",
              "annotations_authorization.k8s.io/reason": "",
              "apiVersion": "audit.k8s.io/v1",
              "auditID": "6b78966f-b699-4965-acc8-xxxxxxxxxxxx",
              "hostname": "example-master-2",
              "k8s_audit_level": "Metadata",
              "kind": "Event",
              "level": "Metadata",
              "log_source": "kubeAPI",
              "log_type": "audit",
              "objectRef_apiVersion": "v1",
      [..]
              "userAgent": "kubelet/v1.29.10+67d3387 (linux/amd64) kubernetes/2fa4795",
              "verb": "watch"
          }
      }

      Vector allows this kind of remapping using VRL: https://vector.dev/docs/reference/configuration/transforms/remap/

      So the feature in this RFE would allow customers to define arbitrary VRL to transform the log messages to the expected format.

      Why does the customer need this? (List the business requirements)

      Splunk licenses apply to the `event` field, but not to the `fields` field. The customer would be able to transform the log message into a format that would result in lower Splunk costs.

      List any affected packages or components.

      Log Collector (Vector)

      Log Forwarding

              jamparke@redhat.com Jamie Parker
              rhn-support-skrenger Simon Krenger
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated: