Connection tracking

The purpose is to provide a way to group flows being part of the same connection, to reduce the noise, ease the navigation in the netflow table and overall make it easier to reason about flows.

Design Doc:

https://docs.google.com/document/d/12rNCWyishTMjKLWKk-Orq68hymcEMqm__ETaLxc59Xg/edit?usp=sharing

The progress it tracked in the GitHub issue:
https://github.com/netobserv/flowlogs-pipeline/issues/241

Connection tracking enable a new stage in FLP to aggregate metrics per "conversation".

A conversation is defined by:

• the grouping of peers identified by their ips, ports and protocols resulting in a unique hashId
• events represented by:
  • newConnection when a unknown connection is starting or TCP flag intercepted
  • heartbeat at every specified interval defined in flow collector spec.processor.connectionUpdateInterval while the connection is up
  • endConnection when flow collector spec.processor.connectionEndTimeout is reached or TCP flag intercepted

The console plugin is slightly adapted to allow the user to query either flows or conversation events:

• new Log type query option (either Conversation or Flow)
• 2 new columns
  • event / type
  • conversation ID
• the ability to filter on a specific conversation ID
• the ability to switch between conversation / flows from side panel keeping the conversation id

The connection tracking is optionnal as the storage will be impacted by these extra logs.

To do so, you can configure it in flow collector as:

• spec.processor.logTypes enum that can be either:
  • FLOWS to export flowLog
  • CONVERSATIONS to export newConnection, heartbeat and endConnection events without related flows
  • ENDED_CONVERSATIONS to export only endConnection events
  • ALL to export newConnection, heartbeat and endConnection events and flowLog
    • this is the most CPU / storage consuming option. We should warn the users in the documentation

The console plugin will disable the "Conversation" query option accordingly.

Config	Plugin