Tracking upstream here: https://github.com/strimzi/strimzi-kafka-operator/issues/11588

Related problem

Cruise Control assumes that each Kafka broker's CPU capacity is fixed and well-defined. However, in containerized environments like Kubernetes, the effective CPU capacity of a broker pod is not necessarily stable or accurate, particularly when:

• CPU requests != CPU limits.
• Pods can burst beyond their requests when limits are higher.
• Cruise Control measures CPU usage based on JVM-visible CPU availability, which may fluctuate over time.
• Cruise Control’s configured CPU capacities do not reflect the actual resources available to the broker at runtime.

This could lead to a mismatch in the actual CPU resource utilization of Kafka brokers and the CPU resource utilization of Kafka brokers reported by Cruise Control. This discrepancy can cause Cruise Control to make poor rebalancing decisions by shifting CPU load based on underestimated or overestimated CPU capacity.

For example, when broker that has a request = 1000m and a limit = 2000m bursts to use 2000m over an extended period of time. Strimzi Cruise Control would report that the broker is using 100% of its configured CPU capacity, 1000m, while it is using 2000m.

Related to #6265

Suggested solution

The best path forward is likely to discuss and define a solution in a proposal.

To start exploring potential directions, here are a few initial ideas worth investigating:

(1) Implement a custom BrokerCapacityConfigResolver plugin [1] that dynamically resolves the resource capacities of broker pods at runtime.

(2) Enhance the Cruise Control metric reporter to report absolute CPU usage (e.g. CPU time or millicores used) rather than relative CPU utilization, which currently depends on the fluctuating effective CPU available to the brokers.

Alternatives

We have discussed and documented a workaround to this issue by setting:

resource.requests.cpu == resource.limits.cpu
 

This workaround reserves the same CPU resources upfront, essentially setting the same CPU capacity for every broker. However, it also:

• Sidelines those CPU resources only to ever be used by the broker, even when those resources are not being used.
• Throttles the broker when it tries to use more CPU resources than allotted, even when there are spare cycles available on the node.

Therefore, it is not viewed as an acceptable long-term solution.

Additional context

[1] https://github.com/linkedin/cruise-control/wiki/Pluggable-Components#broker-capacity-config-resolver