Proposed title of this feature request

Allow log content filtering with Vector "filter" transformation

What is the nature and description of the request?

OpenShift Logging provides features to configuring content filtering with "type: drop" and "type: prune". This is already very helpful for reducing the amount of logs for customers.

However when customers want to filter logs by an "allow-list" rather than an excluding list, having just these two types available is not ideal.

This feature request asks for the implementation of "type: filter" so that the filter transformation of Vector can be configured. This will allow customers to do a "allow-list" approach instead of a "disallow-list" that we have at the moment with the "drop" filter.

Why does the customer need this? (List the business requirements)

Customer needs to filter log to reduce log volume as they have a lot of volume and this results in high costs on the Log Storage side.

Customer use case

Customers want to filter out all low level infrastructure logs but we want to keep the sshd and sudo logs (also the ones with lower level).

Example of Vector "filter" transforms:

      transforms:
        filter_logs:
          type: filter
          inputs:
          - splunk_hec_input
          condition: '.log_type == "infrastructure" && (includes(["sshd", "sudo"],.systemd.u.SYSLOG_IDENTIFIER) || !includes(["default", "notice", "debug", "info", "trace"],.level))'

Possible solution with "drop" filter:

Assumption: "if there is an error evaluating a condition, for example, the field is missing from the log record being evaluated, that condition evaluates to false".

kind: ClusterLogForwarder
metadata:
# ...
spec:
  filters:
  - name: important
    type: drop
    drop:
      test:
      - field: .log_type
        matches: "infrastructure"
      - field: .systemd.u.SYSLOG_IDENTIFIER
        notMatches: "sshd|sudo"
      - field: .level
        matches: "default|notice|debug|info|trace"
      test:
      - field: .log_type
        matches: "infrastructure"
      - field: .level
        matches: "default|notice|debug|info|trace"

Desired solution with a "filter" filter:

kind: ClusterLogForwarder
metadata:
# ...
spec:
  filters:
  - name: important
    type: filter
    filter:
      test:
      - field: .log_type
        matches: "infrastructure"
      - field: .systemd.u.SYSLOG_IDENTIFIER
        matches: "sshd|sudo"
      test:
      - field: .log_type
        matches: "infrastructure"
      - field: .level
        notMatches: "default|notice|debug|info|trace"

List any affected packages or components.

Log Collection / Vector