At the moment, we use Kafka Bridge to:

• receive messages that were sent by our SWATCH services
• send messages to topics

The problem is that to receive/send messages, we need to create a new consumer instance and a consumer group using the API that Kafka Bridge exposes, and then Kafka needs to reconcile these consumers which is causing flakiness to the component tests.

Note that we can't directly run commands to execute the Kafka scripts in the Kafka pod because we don't have privileges to do so in an ephemeral environment.

Some alternatives are:

• Expose a custom API in our services (only in DEV mode and in ephemeral) to send a message to a kafka topic / to get all the messages that the service sends.
• Speak with Clowder team to allow run commands to the kafka component

Acceptance Criteria

• Investigate alternatives to Kafka Bridge to the use cases mentioned above.
• Document with the analysis and share it with the team