There is an frequent(almost on every run) intermittent failure on one of my integration tests[1].

For the current release I found a workaround for it, by fixing the workers thread pool in Netty transport to 1. Just rollback my commit on NettyTransport (and re- mvn install server/core, I always use to forget that ) From my investigation, I think the problem is caused by the following scenario:

• AbstractProtocolDecoder.decode is being called upon a request from client, the code is being processed by worker thread ONE
• ONE reads the header, but no more information is available and it blocks on some read (TCP splits the request from server into multiple packets and NIO delivers them in multiple calls)
• AbstractProtocolDecoder.decode is being called again, with the rest of bytes from the same request, but within thread TWO
• thread TWO tries to read magic (skips bytes) and thread ONE hangs, waiting for more information.

If this is the problem , a solution would be to confine each channel to a single processing thread, as netty might use multiple processing threads for bytes originating from the same channel.

[1] [trunk/client/hotrod-client ]$ mvn test -Dtest=HotRodIntegrationTest

p.s. This might affect memcached as well, double check.