Infinispan's default KeyAffinityService implementation has a number of problems:
- Its topology change listener is blocking. This can cause deadlocks in recent Infinispan releases, which requires listeners to be non-blocking.
- Its getKeyForAddress(...) method throws an ISE if the target address does not own any segments. This can be the case if state transfer fails, or if a node is configured with zero capacity. We don't currently handle this IAE. Ideally, this condition should generate a random key.
- It uses a single worker thread to generate keys for interested members. Consequently, it must offer keys with a timeout, lest it starve generation of keys for other members. This causes results in overly high CPU utilization when all queues are full.
- getKeyForAddress(...) blocks during topology changes, which ultimately blocks request threads while Infinispan repopulates its key queues.
- It flushes it key queues unnecessarily during intermediate state transfer phases.
Given that EAP's use case for KeyAffinityService only cares about generating keys that hash locally, we can create a custom implementation optimized for our use case.