Infinispan's default KeyAffinityService implementation has a number of problems:
- Its topology change listener is blocking. This can cause deadlocks in recent Infinispan releases, which requires listeners to be non-blocking.
- Its getKeyForAddress(...) method throws an ISE if the target address does not own any segments. This can be the case if state transfer fails, or if a node is configured with zero capacity. We don't currently handle this IAE. Ideally, this condition should generate a random key.
- It uses a single worker thread to generate keys for interested members. Consequently, it must offer keys with a timeout, lest it starve generation of keys for other members. This causes results in overly high CPU utilization when all queues are full.
- getKeyForAddress(...) blocks during topology changes, which ultimately blocks request threads while Infinispan repopulates its key queues.
- It flushes it key queues unnecessarily during intermediate state transfer phases.
In order to fix some of these issues, we will copy Infinispan's implementation into EAP 7.3.x, and patch as needed.
7.4.x and upstream will use a completely new implementation optimized for our use case.