One reason Cruise Control struggles to maintain compatibility across multiple Kafka versions is its reliance on unstable internal Kafka APIs. By migrating to public Kafka APIs instead of the internal ones, we can greatly reduce the complexity of code contributions and expand the Kafka version coverage. However, implementing the necessary changes is first likely to not be trivial (we would probably need to build a test suite with testcontainers and the Apache Kafka image but avoid internal APIs) and relies on LinkedIn accepting all code changes.

This task involves:

• Investigating where internal APIs are being used in the upstream Cruise Control code base. These should be summarised in a document and proposals for moving to public APIs made.
• We should also come up with a proposal around improvements to the testing setup to decouple from specific Kafka library versions. Testcontainers is one option here. Making it possible to test against a defined range of Kafka versions would be another useful addition.
• Once this document is reviewed by the team it should be proposed in the Ideas section of the upstream Cruise Control GitHub Discussion site. As CC doesn't seem to have a formal proposal process this is a way for us to get comment and hopefully buy-in from the maintainers.
• Once we have a resolution upstream we should divide the migration of internal API usage into smaller tasks/contributions that can be easily implemented, tested, reviewed, and accepted in upstream Cruise Control.