we're occasionally getting stale data on remote EJB invocations (number counter returns number 1 lower than expected, see example).
This is usually preceded (~6 seconds before that) by cluster wide rebalance after a node is brought back from dead.
- 2000 clients, stale data is uncommon
- requests from a single client are separated by a 4 second window.
An example of stale data:
And a link to our jobs if you're interested:
This behavior has so far been observed with jvmkill and undeploy scenario, on REPL-SYNC and DIST-SYNC caches.