WHAT

In Kafka 3.4.0, we plan to have the following new metrics:
KIP-847 exposes a metric to get the producerId counts. Too many producer Id created will cause broker OOM since we store the producerId and its metadata in memory.

WHY

In idempotent producers, a new producer id is created when KafkaProducer is created. A badly written application may frequently create new KafkaProducer objects. This is not optimal in general, but specifically for idempotent producers, doing so would pollute broker memory with producer ids and related metadata. Even though the metadata for each producer id is small, creating too many producer ids could run brokers out of memory.

HOW

<Suggestions for how this may be solved.> [Optional]

DONE

1. Add a new alert when producerIDCount metric goes too high
2. Add a SOP for the alert