what we'd like to do:
Filter logs by severity level
Reduce metadata
Split CNF logs out and send to different Kafka topic

Since the operator cannot do this today, the agreement was to set the log collector to unmanaged and edit the vector.toml.... - we can update the vector config to meet the requirements without having to unmanage the logging stack.  We prefer to change as little as possible.

Here's my diff:

# oc get secret -n openshift-logging collector-config -o jsonpath='{.data.vector\.toml}' | base64 -d  | diff vector.toml -
254,255c254
< route.app = '!((starts_with!(.kubernetes.namespace_name,"kube-")) || (starts_with!(.kubernetes.namespace_name,"openshift-")) || (.kubernetes.namespace_name == "default") || (.kubernetes.namespace_name == "openshift") || (.kubernetes.namespace_name == "kube") || (exists(.kubernetes.namespace_labels.fullName)))'
< route.cnf = 'exists(.kubernetes.namespace_labels.fullName)'
---
> route.app = '!((starts_with!(.kubernetes.namespace_name,"kube-")) || (starts_with!(.kubernetes.namespace_name,"openshift-")) || (.kubernetes.namespace_name == "default") || (.kubernetes.namespace_name == "openshift") || (.kubernetes.namespace_name == "kube"))'
266,273d264
< # Set log_type to "cnf"
< [transforms.cnf]
< type = "remap"
< inputs = ["route_container_logs.cnf"]
< source = '''
<   .log_type = "cnf"
< '''
<
292,343d282
< [transforms.isolate-cnf]
< type = "remap"
< inputs = ["cnf"]
< source = '''
<  .fullName = .kubernetes.namespace_labels.fullName
<   del(.kubernetes.namespace_labels)
<   del(.kubernetes.labels)
<   del(.kubernetes.annotations)
<   del(.kubernetes.pod_id)
<   del(.host.mac)
<   del(.host.ip)
<   del(.objectRef.apiVersion)
<   del(.objectRef.apiGroup)
<   del(.user.extra)
<   del(.user.uid)
<   del(.userAgent)
<   del(.openshift)
<
<   .openshift.labels = {"acmname":"mgmt-hub-2","cluster":"sno-11","clustertype":"sno","datacenter":"wnckcaau"}
< '''
<
< [transforms.clean-labels]
< type = "remap"
< inputs = ["application", "infrastructure"]
< source = '''
<   .fullName = .kubernetes.namespace_labels.fullName
<   del(.kubernetes.namespace_labels)
<   del(.kubernetes.labels)
<   del(.kubernetes.annotations)
<   del(.kubernetes.pod_id)
<   del(.host.mac)
<   del(.host.ip)
<   del(.objectRef.apiVersion)
<   del(.objectRef.apiGroup)
<   del(.user.extra)
<   del(.user.uid)
<   del(.userAgent)
<   del(.openshift)
< '''
<
< [transforms.app-infra-filtered]
< type = "filter"
< inputs = ["clean-labels"]
< condition = ".level != \"info\" && .level != \"debug\" && .level != \"notice\""
<
< [transforms.clean-audit]
< type = "remap"
< inputs =  ["audit"]
< source = '''
<   #map_keys(.) -> |key| { if !(match_array( ["@timestamp","hostname","verb","log_type","objectRef","auditID","user","file"], key)) { del(key) } }
< '''
<
346c285
< inputs = ["clean-audit","app-infra-filtered"]
---
> inputs = ["application","audit","infrastructure"]
364d302
<
371,390d308
< # Kafka config for CNF applications
< [sinks.kafka_cnf]
< type = "kafka"
< inputs = ["isolate-cnf"]
< bootstrap_servers = "batman.hqplan.lab:30287"
< topic = "fe-cnf-logs"
<
< [sinks.kafka_cnf.encoding]
< codec = "json"
< timestamp_format = "rfc3339"
<
< [sinks.kafka_cnf.librdkafka_options]
< "enable.ssl.certificate.verification" = "false"
<
< [sinks.kafka_cnf.tls]
< enabled = true
< key_file = "/var/run/ocp-collector/secrets/kafka-secret/tls.key"
< crt_file = "/var/run/ocp-collector/secrets/kafka-secret/tls.crt"
< ca_file = "/var/run/ocp-collector/secrets/kafka-secret/ca-bundle.crt"
<
407c325
< crt_file = "/etc/collector/metrics/tls.crt"
---
> crt_file = "/etc/collector/metrics/tls.crt"
\ No newline at end of file

I can share the actual code...   I do have one part that is not working yet...  the clean-audit transform - that map_keys function doesn't work yet - looks like it matches docs, so maybe its a vector version thing - or I just don't understand the docs syntax

My method to separate CNF logs from other application logs is to look for the fullName label - so that assumption may need to be adjusted.  There's probably other ways to do the same thing.

My goal here was not to optimize the code - once we can measure the cpu impact, we can do so - I suspect we can move the large filters to earlier sections to reduce cpu overall.

Here's some commands to do this manually:

# implement
vi vector.toml
oc patch ClusterLogging instance -n openshift-logging --type merge -p '{"spec": {"managementState": "Unmanaged"}}'
oc create secret generic -n openshift-logging collector-config --dry-run=client -o yaml --from-file=vector.toml | oc replace -f -
oc delete pod -n openshift-logging -l component=collector

# monitor
oc logs -n openshift-logging -l component=collector --follow -c collector

# revert
oc patch ClusterLogging instance -n openshift-logging --type merge -p '{"spec": {"managementState": "Managed"}}'
oc delete pod -n openshift-logging -l component=collector