Hi,

we're occasionally getting stale data on remote EJB invocations (number counter returns number 1 lower than expected, see example). This is usually preceded (~6 seconds before that) by cluster wide rebalance after a node is brought back from dead.

• 2000 clients, stale data is uncommon
• requests from a single client are separated by a 4 second window.

An example of stale data:

2015/08/28 13:01:35:442 EDT [WARN ][Runner - 640] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Error sampling data:  <org.jboss.smartfrog.loaddriver.RequestProcessingException: Response serial does not match. Expected: 86, received: 85, runner: 640.>

And a link to our jobs if you're interested:
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-failover-ejb-ejbremote-jvmkill-repl-sync/5/console-perf17/consoleText

This behavior has so far been observed with jvmkill and undeploy scenario, on REPL-SYNC and DIST-SYNC caches.