NETOBSERV-640 Support for Export of Enriched NetFlow Data


Installed kfk - kafka CLI

brew install strimzi-kafka-cli


$ kfk --version
CLI Version: 0.1.0a64
Strimzi Version: 0.31.1
Kubectl Version: v1.25.2

Deployed Kafka cluster, KafkaTopic with AMQ Streams Operator and also running flowcollector in KAFKA mode.

Created topic to export flows:

$ cat << EOF | oc create -f -
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: netobserv-flows-export
  labels:
    strimzi.io/cluster: "kafka-cluster"
spec:
  partitions: 24
  replicas: 1
EOF


Listed topics:

$ kfk topics --cluster kafka-cluster -n netobserv  --list
NAME                                                                                               CLUSTER         PARTITIONS   REPLICATION FACTOR   READY
consumer-offsets---84e7a678d08f4bd226872e5cdd4eb527fadc1c6a                                        kafka-cluster   50           1                    True
netobserv-flows-export                                                                             kafka-cluster   24           1                    True
network-flows                                                                                      kafka-cluster   10           1                    True
strimzi-store-topic---effb8e3e057afce1ecf67c3f5d8e4e3ff177fc55                                     kafka-cluster   1            3                    True
strimzi-topic-operator-kstreams-topic-store-changelog---b75e702040b99be8a9263134de3507fc0cc4017b   kafka-cluster   1            1                    True





Edited flowcollector to add exporters:

$ oc get flowcollector -o yaml -o jsonpath='{.items[].spec.exporters}' | jq
[
  {
    "kafka": {
      "address": "kafka-cluster-kafka-bootstrap.netobserv",
      "topic": "netobserv-flows-export"
    },
    "type": "KAFKA"
  }
]




$ oc get pod -l app=flowlogs-pipeline-transformer
NAME                                             READY   STATUS    RESTARTS   AGE
flowlogs-pipeline-transformer-5f6576c755-ffcd2   1/1     Running   0          25s
flowlogs-pipeline-transformer-5f6576c755-v8nst   1/1     Running   0          14s
flowlogs-pipeline-transformer-5f6576c755-w4scq   1/1     Running   0          36s




$ kfk console-consumer --topic network-flows-export  -n netobserv -c kafka-cluster --from-beginning
{"Bytes":622,"DstAddr":"10.0.129.209","DstK8S_Name":"ip-10-0-129-209.us-east-2.compute.internal","DstK8S_OwnerName":"ip-10-0-129-209.us-east-2.compute.internal","DstK8S_OwnerType":"Node","DstK8S_Type":"Node","DstMac":"06:71:7F:99:DC:DA","DstPort":6081,"Duplicate":false,"Etype":2048,"FlowDirection":1,"Interface":"br-ex","Packets":3,"Proto":17,"SrcAddr":"10.0.160.46","SrcK8S_Name":"ip-10-0-160-46.us-east-2.compute.internal","SrcK8S_OwnerName":"ip-10-0-160-46.us-east-2.compute.internal","SrcK8S_OwnerType":"Node","SrcK8S_Type":"Node","SrcMac":"06:F6:81:E3:CA:2E","SrcPort":17372,"TimeFlowEndMs":1673547283869,"TimeFlowStartMs":1673547283867,"TimeReceived":1673547313}
{"Bytes":257,"DstAddr":"10.0.190.160","DstK8S_Name":"ip-10-0-190-160.us-east-2.compute.internal","DstK8S_OwnerName":"ip-10-0-190-160.us-east-2.compute.internal","DstK8S_OwnerType":"Node","DstK8S_Type":"Node","DstMac":"02:A7:4A:84:2E:CC","DstPort":2379,"Duplicate":false,"Etype":2048,"FlowDirection":1,"Interface":"br-ex","Packets":3,"Proto":6,"SrcAddr":"10.0.131.213","SrcK8S_Name":"ip-10-0-131-213.us-east-2.compute.internal","SrcK8S_OwnerName":"ip-10-0-131-213.us-east-2.compute.internal","SrcK8S_OwnerType":"Node","SrcK8S_Type":"Node","SrcMac":"02:5B:6E:E4:B0:AC","SrcPort":59828,"TimeFlowEndMs":1673547279852,"TimeFlowStartMs":1673547279810,"TimeReceived":1673547313}
{"Bytes":132,"DstAddr":"10.0.129.209","DstK8S_Name":"ip-10-0-129-209.us-east-2.compute.internal","DstK8S_OwnerName":"ip-10-0-129-209.us-east-2.compute.internal","DstK8S_OwnerType":"Node","DstK8S_Type":"Node","DstMac":"06:71:7F:99:DC:DA","DstPort":6081,"Duplicate":true,"Etype":2048,"FlowDirection":1,"Interface":"br-ex","Packets":1,"Proto":17,"SrcAddr":"10.0.160.46","SrcK8S_Name":"ip-10-0-160-46.us-east-2.compute.internal","SrcK8S_OwnerName":"ip-10-0-160-46.us-east-2.compute.internal","SrcK8S_OwnerType":"Node","SrcK8S_Type":"Node","SrcMac":"06:F6:81:E3:CA:2E","SrcPort":49795,"TimeFlowEndMs":1673547282485,"TimeFlowStartMs":1673547282485,"TimeReceived":1673547313}


Verified flows are exported to netobserv-flows-export KafkaTopic.

Other tests tried successfully:
1. Removing exporter.
2. Incorrect exporter topic. 

FLP handled well in above scenarios.

Measured performance impact with node-density-heavy workload - found higher CPU utilization than without exporter but still within limits.