During load testing of 5.1.0.CR1, we're encountering JGroups 3.x dropping views. We know due to ISPN-1581, if the number of view changes > 3, there could be a stale lock on a failed commit. However, we're seeing stale locks occur on a single view change.

In the following logs, the affected cluster is the erm-cluster-xxxx

(We also don't know why JGroups 3.x is unstable. We suspected FLUSH and incorrect FD settings, but we removed them, and we're still getting dropped messages)

The trace logs (It isn't long at all before the issue occurs) are at:

http://dl.dropbox.com/u/50401510/5.1.0.CR1/dec08viewchange/dht10/server.log.gz
http://dl.dropbox.com/u/50401510/5.1.0.CR1/dec08viewchange/dht11/server.log.gz
http://dl.dropbox.com/u/50401510/5.1.0.CR1/dec08viewchange/dht12/server.log.gz
http://dl.dropbox.com/u/50401510/5.1.0.CR1/dec08viewchange/dht13/server.log.gz
http://dl.dropbox.com/u/50401510/5.1.0.CR1/dec08viewchange/dht14/server.log.gz
http://dl.dropbox.com/u/50401510/5.1.0.CR1/dec08viewchange/dht15/server.log.gz