If a tenant suddenly starts sending us too many (millions of) Kafka messages on the platform.notifications.ingress topic for a wrong reason, that could delay the processing of the Kafka messages from all the other tenants.

Today, we don't have any kind of rate limiting or the ability to temporarily blacklist a tenant. Rate limiting might not be the right approach because some tenants (RBAC in particular) sometimes need to send us millions of messages at once and that is perfectly normal (RBAC seeding). Blacklisting on the other hand could work with all tenants. Blacklisting would likely rely on a variant in Unleash where we would list the blacklisted tenants. But we would then need to decide how to deal with messages when they come from a blacklisted tenant. They could be:

• ignored (not stored in the DB)
• stored in the DB but not processed further
• forwarded to a different Kafka topic with a lower priority/throughput
• [insert here another idea that makes sense]

We need something to protect all tenants from a potential bad neighbor. This spike will be used to determine how to do that.