Use case

User wants sensitive field values in logs to be replaced with a "redaction" marker, such as the string "REDACTED", while keeping the field keys in the log. This hides the sensitive data, but allows the user to see that the field was present in the original log. Example:

   hostname: host.on.secret.network 

replaced with:

 hostname: REDACTED  

API

The exisitng 'prune' filter does something similar, except that it completely removes the field key and value rather than replacing the value with a REDACTED marker.

Add a new filter called 'redact', which has the same configuration as the prune filter with the additional field:

    redactMarker: string # String to use for redaction, default "REDACT" 

Implementation

Should re-use the Go and VRL code for the 'prune' filter for the common part of choosing which fields to prune or redact.

Open questions

A string like "REDACTED" may not be a valid value for all fields in a log record.

Downstream log processors may have problems processing log records with invalid "REDACTED" values.

For example:

• FIelds that are expected to have numeric, boolean, object or array values.
• Fields with restricted string values like enums or pattern validation.

Workarounds:

1. Ignore the problem, leave error handling to the receiver.
2. Use multiple "redact" filters with different "redactMarker" values for each type of field that needs a different marker.
3. Combine the "redact" filter with  a "prune" filter to prune fields that can't be redacted safely.