2011-02-23 11:23:16,061 INFO [org.infinispan.server.core.Main$] (main) Start main with args: {0} 2011-02-23 11:23:16,959 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (InfinispanServer-Main) Starting JGroups Channel 2011-02-23 11:23:17,306 WARN [org.jgroups.protocols.UDP] (InfinispanServer-Main) send buffer of socket java.net.DatagramSocket@30d83d was set to 640KB, but the OS only allocated 131.07KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux) 2011-02-23 11:23:17,306 WARN [org.jgroups.protocols.UDP] (InfinispanServer-Main) receive buffer of socket java.net.DatagramSocket@30d83d was set to 20MB, but the OS only allocated 131.07KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux) 2011-02-23 11:23:17,306 WARN [org.jgroups.protocols.UDP] (InfinispanServer-Main) send buffer of socket java.net.MulticastSocket@c4afc4 was set to 640KB, but the OS only allocated 131.07KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux) 2011-02-23 11:23:17,306 WARN [org.jgroups.protocols.UDP] (InfinispanServer-Main) receive buffer of socket java.net.MulticastSocket@c4afc4 was set to 25MB, but the OS only allocated 131.07KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux) 2011-02-23 11:23:20,321 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (InfinispanServer-Main) Received new cluster view: [localhost-42248|0] [localhost-42248] 2011-02-23 11:23:20,329 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (InfinispanServer-Main) Cache local address is localhost-42248, physical addresses are [10.200.130.82:55387] 2011-02-23 11:24:16,418 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-1,localhost-42248) Received new cluster view: [localhost-42248|1] [localhost-42248, localhost-43522] 2011-02-23 11:24:16,422 INFO [org.infinispan.distribution.DistributionManagerImpl] (Incoming-1,localhost-42248) Detected a view change. Member list changed from [localhost-42248] to [localhost-42248, localhost-43522] 2011-02-23 11:24:16,422 INFO [org.infinispan.distribution.DistributionManagerImpl] (Incoming-1,localhost-42248) This is a JOIN event! Wait for notification from new joiner localhost-43522 2011-02-23 11:25:21,820 ERROR [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (STREAMING_STATE_TRANSFER-sender-1,localhost-42248) Caught while responding to state transfer request org.infinispan.statetransfer.StateTransferException: java.util.concurrent.TimeoutException: STREAMING_STATE_TRANSFER-sender-1,localhost-42248 could not obtain exclusive processing lock after 10 seconds. Locks in question are java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock@1b35ece[Read locks = 1] and java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock@115470e[Unlocked] at org.infinispan.statetransfer.StateTransferManagerImpl.generateState(StateTransferManagerImpl.java:176) at org.infinispan.remoting.InboundInvocationHandlerImpl.generateState(InboundInvocationHandlerImpl.java:217) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.getState(JGroupsTransport.java:565) at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUpEvent(MessageDispatcher.java:691) at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:772) at org.jgroups.JChannel.up(JChannel.java:1465) at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:954) at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER$StateProviderHandler.process(STREAMING_STATE_TRANSFER.java:653) at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER$StateProviderThreadSpawner$1.run(STREAMING_STATE_TRANSFER.java:582) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.util.concurrent.TimeoutException: STREAMING_STATE_TRANSFER-sender-1,localhost-42248 could not obtain exclusive processing lock after 10 seconds. Locks in question are java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock@1b35ece[Read locks = 1] and java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock@115470e[Unlocked] at org.infinispan.remoting.transport.jgroups.JGroupsDistSync.acquireProcessingLock(JGroupsDistSync.java:79) at org.infinispan.statetransfer.StateTransferManagerImpl.generateTransactionLog(StateTransferManagerImpl.java:203) at org.infinispan.statetransfer.StateTransferManagerImpl.generateState(StateTransferManagerImpl.java:166) ... 11 more 2011-02-23 11:28:56,554 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (HotRodServerWorker-1-2) Execution error: org.infinispan.util.concurrent.TimeoutException: Unable to acquire lock after [10 seconds] on key [view] for requestor [Thread[HotRodServerWorker-1-2,5,main]]! Lock held by [(another thread)] at org.infinispan.container.EntryFactoryImpl.acquireLock(EntryFactoryImpl.java:228) at org.infinispan.container.EntryFactoryImpl.wrapEntryForWriting(EntryFactoryImpl.java:155) at org.infinispan.container.EntryFactoryImpl.wrapEntryForWriting(EntryFactoryImpl.java:107) at org.infinispan.interceptors.LockingInterceptor.visitReplaceCommand(LockingInterceptor.java:337) at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:63) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) at org.infinispan.commands.AbstractVisitor.visitReplaceCommand(AbstractVisitor.java:66) at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:63) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:184) at org.infinispan.interceptors.TxInterceptor.visitReplaceCommand(TxInterceptor.java:142) at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:63) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) at org.infinispan.commands.AbstractVisitor.visitReplaceCommand(AbstractVisitor.java:66) at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:63) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:95) at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:62) at org.infinispan.commands.AbstractVisitor.visitReplaceCommand(AbstractVisitor.java:66) at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:63) at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:273) at org.infinispan.CacheDelegate.replace(CacheDelegate.java:482) at org.infinispan.CacheSupport.replace(CacheSupport.java:96) at org.infinispan.server.hotrod.HotRodEncoder.writeHashTopologyHeader(HotRodEncoder.scala:180) at org.infinispan.server.hotrod.HotRodEncoder.writeHeader(HotRodEncoder.scala:117) at org.infinispan.server.hotrod.HotRodEncoder.encode(HotRodEncoder.scala:32) at org.infinispan.server.core.transport.netty.EncoderAdapter.encode(EncoderAdapter.scala:17) at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:66) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:568) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:563) at org.jboss.netty.channel.Channels.write(Channels.java:611) at org.jboss.netty.channel.Channels.write(Channels.java:578) at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:259) at org.infinispan.server.core.transport.netty.ChannelAdapter.write(ChannelAdapter.scala:21) at org.infinispan.server.core.AbstractProtocolDecoder.writeResponse(AbstractProtocolDecoder.scala:87) at org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:51) at org.infinispan.server.core.transport.netty.DecoderAdapter.decode(DecoderAdapter.scala:18) at org.infinispan.server.core.transport.netty.DecoderAdapter.decode(DecoderAdapter.scala:14) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:471) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:444) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:545) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:540) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:350) at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:281) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:201) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2011-02-23 11:28:56,562 ERROR [org.infinispan.server.core.AbstractProtocolDecoder$] (HotRodServerWorker-1-2) Exception reported org.infinispan.util.concurrent.TimeoutException: Unable to acquire lock after [10 seconds] on key [view] for requestor [Thread[HotRodServerWorker-1-2,5,main]]! Lock held by [(another thread)] at org.infinispan.container.EntryFactoryImpl.acquireLock(EntryFactoryImpl.java:228) at org.infinispan.container.EntryFactoryImpl.wrapEntryForWriting(EntryFactoryImpl.java:155) at org.infinispan.container.EntryFactoryImpl.wrapEntryForWriting(EntryFactoryImpl.java:107) at org.infinispan.interceptors.LockingInterceptor.visitReplaceCommand(LockingInterceptor.java:337) at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:63) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) at org.infinispan.commands.AbstractVisitor.visitReplaceCommand(AbstractVisitor.java:66) at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:63) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:184) at org.infinispan.interceptors.TxInterceptor.visitReplaceCommand(TxInterceptor.java:142) at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:63) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) at org.infinispan.commands.AbstractVisitor.visitReplaceCommand(AbstractVisitor.java:66) at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:63) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:95) at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:62) at org.infinispan.commands.AbstractVisitor.visitReplaceCommand(AbstractVisitor.java:66) at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:63) at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:273) at org.infinispan.CacheDelegate.replace(CacheDelegate.java:482) at org.infinispan.CacheSupport.replace(CacheSupport.java:96) at org.infinispan.server.hotrod.HotRodEncoder.writeHashTopologyHeader(HotRodEncoder.scala:180) at org.infinispan.server.hotrod.HotRodEncoder.writeHeader(HotRodEncoder.scala:117) at org.infinispan.server.hotrod.HotRodEncoder.encode(HotRodEncoder.scala:32) at org.infinispan.server.core.transport.netty.EncoderAdapter.encode(EncoderAdapter.scala:17) at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:66) at org.jboss.netty.channel.Channels.write(Channels.java:611) at org.jboss.netty.channel.Channels.write(Channels.java:578) at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:259) at org.infinispan.server.core.transport.netty.ChannelAdapter.write(ChannelAdapter.scala:21) at org.infinispan.server.core.AbstractProtocolDecoder.writeResponse(AbstractProtocolDecoder.scala:87) at org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:51) at org.infinispan.server.core.transport.netty.DecoderAdapter.decode(DecoderAdapter.scala:18) at org.infinispan.server.core.transport.netty.DecoderAdapter.decode(DecoderAdapter.scala:14) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:471) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:444) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:350) at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:281) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:201) at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2011-02-23 11:28:56,567 ERROR [org.infinispan.server.core.AbstractProtocolDecoder$] (HotRodServerWorker-1-2) Exception reported java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcher.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:202) at sun.nio.ch.IOUtil.read(IOUtil.java:169) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:243) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:322) at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:281) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:201) at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2011-02-23 11:28:56,567 ERROR [org.infinispan.server.core.AbstractProtocolDecoder$] (HotRodServerWorker-1-2) Exception reported java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:72) at sun.nio.ch.IOUtil.write(IOUtil.java:28) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$PooledSendBuffer.transferTo(SocketSendBufferPool.java:239) at org.jboss.netty.channel.socket.nio.NioWorker.write0(NioWorker.java:470) at org.jboss.netty.channel.socket.nio.NioWorker.writeFromUserCode(NioWorker.java:388) at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:137) at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:76) at org.jboss.netty.channel.Channels.write(Channels.java:632) at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:70) at org.jboss.netty.channel.Channels.write(Channels.java:611) at org.jboss.netty.channel.Channels.write(Channels.java:578) at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:259) at org.infinispan.server.core.transport.netty.ChannelAdapter.write(ChannelAdapter.scala:21) at org.infinispan.server.core.AbstractProtocolDecoder.exceptionCaught(AbstractProtocolDecoder.scala:168) at org.infinispan.server.core.transport.netty.DecoderAdapter.exceptionCaught(DecoderAdapter.scala:22) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:432) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:332) at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:281) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:201) at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2011-02-23 11:28:56,573 ERROR [org.infinispan.server.core.AbstractProtocolDecoder$] (HotRodServerWorker-1-2) Exception reported java.nio.channels.ClosedChannelException at org.jboss.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:637) at org.jboss.netty.channel.socket.nio.NioWorker.close(NioWorker.java:593) at org.jboss.netty.channel.socket.nio.NioWorker.write0(NioWorker.java:513) at org.jboss.netty.channel.socket.nio.NioWorker.writeFromUserCode(NioWorker.java:388) at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:137) at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:76) at org.jboss.netty.channel.Channels.write(Channels.java:632) at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:70) at org.jboss.netty.channel.Channels.write(Channels.java:611) at org.jboss.netty.channel.Channels.write(Channels.java:578) at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:259) at org.infinispan.server.core.transport.netty.ChannelAdapter.write(ChannelAdapter.scala:21) at org.infinispan.server.core.AbstractProtocolDecoder.exceptionCaught(AbstractProtocolDecoder.scala:168) at org.infinispan.server.core.transport.netty.DecoderAdapter.exceptionCaught(DecoderAdapter.scala:22) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:432) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:332) at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:281) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:201) at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2011-02-23 11:29:00,784 ERROR [org.infinispan.remoting.rpc.RpcManagerImpl] (HotRodServerWorker-1-1) unexpected error while replicating java.util.concurrent.TimeoutException: Timed out waiting for a cluster-wide sync to be released. (timeout = 240 seconds) at org.infinispan.remoting.transport.jgroups.JGroupsDistSync.blockUntilReleased(JGroupsDistSync.java:54) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:390) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:102) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:126) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:231) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:218) at org.infinispan.remoting.rpc.RpcManagerImpl.broadcastRpcCommand(RpcManagerImpl.java:201) at org.infinispan.remoting.rpc.RpcManagerImpl.broadcastRpcCommand(RpcManagerImpl.java:194) at org.infinispan.interceptors.ReplicationInterceptor.handleCrudMethod(ReplicationInterceptor.java:115) at org.infinispan.interceptors.ReplicationInterceptor.visitReplaceCommand(ReplicationInterceptor.java:98) at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:63) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.LockingInterceptor.visitReplaceCommand(LockingInterceptor.java:338) at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:63) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) at org.infinispan.commands.AbstractVisitor.visitReplaceCommand(AbstractVisitor.java:66) at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:63) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:184) at org.infinispan.interceptors.TxInterceptor.visitReplaceCommand(TxInterceptor.java:142) at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:63) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) at org.infinispan.commands.AbstractVisitor.visitReplaceCommand(AbstractVisitor.java:66) at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:63) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:95) at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:62) at org.infinispan.commands.AbstractVisitor.visitReplaceCommand(AbstractVisitor.java:66) at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:63) at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:273) at org.infinispan.CacheDelegate.replace(CacheDelegate.java:482) at org.infinispan.CacheSupport.replace(CacheSupport.java:96) at org.infinispan.server.hotrod.HotRodEncoder.writeHashTopologyHeader(HotRodEncoder.scala:180) at org.infinispan.server.hotrod.HotRodEncoder.writeHeader(HotRodEncoder.scala:117) at org.infinispan.server.hotrod.HotRodEncoder.encode(HotRodEncoder.scala:32) at org.infinispan.server.core.transport.netty.EncoderAdapter.encode(EncoderAdapter.scala:17) at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:66) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:568) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:563) at org.jboss.netty.channel.Channels.write(Channels.java:611) at org.jboss.netty.channel.Channels.write(Channels.java:578) at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:259) at org.infinispan.server.core.transport.netty.ChannelAdapter.write(ChannelAdapter.scala:21) at org.infinispan.server.core.AbstractProtocolDecoder.writeResponse(AbstractProtocolDecoder.scala:87) at org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:51) at org.infinispan.server.core.transport.netty.DecoderAdapter.decode(DecoderAdapter.scala:18) at org.infinispan.server.core.transport.netty.DecoderAdapter.decode(DecoderAdapter.scala:14) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:471) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:444) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:545) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:540) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:350) at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:281) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:201) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2011-02-23 11:29:00,787 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (HotRodServerWorker-1-1) Execution error: org.infinispan.CacheException: java.util.concurrent.TimeoutException: Timed out waiting for a cluster-wide sync to be released. (timeout = 240 seconds) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:115) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:126) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:231) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:218) at org.infinispan.remoting.rpc.RpcManagerImpl.broadcastRpcCommand(RpcManagerImpl.java:201) at org.infinispan.remoting.rpc.RpcManagerImpl.broadcastRpcCommand(RpcManagerImpl.java:194) at org.infinispan.interceptors.ReplicationInterceptor.handleCrudMethod(ReplicationInterceptor.java:115) at org.infinispan.interceptors.ReplicationInterceptor.visitReplaceCommand(ReplicationInterceptor.java:98) at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:63) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.LockingInterceptor.visitReplaceCommand(LockingInterceptor.java:338) at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:63) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) at org.infinispan.commands.AbstractVisitor.visitReplaceCommand(AbstractVisitor.java:66) at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:63) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:184) at org.infinispan.interceptors.TxInterceptor.visitReplaceCommand(TxInterceptor.java:142) at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:63) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) at org.infinispan.commands.AbstractVisitor.visitReplaceCommand(AbstractVisitor.java:66) at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:63) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:95) at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:62) at org.infinispan.commands.AbstractVisitor.visitReplaceCommand(AbstractVisitor.java:66) at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:63) at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:273) at org.infinispan.CacheDelegate.replace(CacheDelegate.java:482) at org.infinispan.CacheSupport.replace(CacheSupport.java:96) at org.infinispan.server.hotrod.HotRodEncoder.writeHashTopologyHeader(HotRodEncoder.scala:180) at org.infinispan.server.hotrod.HotRodEncoder.writeHeader(HotRodEncoder.scala:117) at org.infinispan.server.hotrod.HotRodEncoder.encode(HotRodEncoder.scala:32) at org.infinispan.server.core.transport.netty.EncoderAdapter.encode(EncoderAdapter.scala:17) at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:66) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:568) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:563) at org.jboss.netty.channel.Channels.write(Channels.java:611) at org.jboss.netty.channel.Channels.write(Channels.java:578) at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:259) at org.infinispan.server.core.transport.netty.ChannelAdapter.write(ChannelAdapter.scala:21) at org.infinispan.server.core.AbstractProtocolDecoder.writeResponse(AbstractProtocolDecoder.scala:87) at org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:51) at org.infinispan.server.core.transport.netty.DecoderAdapter.decode(DecoderAdapter.scala:18) at org.infinispan.server.core.transport.netty.DecoderAdapter.decode(DecoderAdapter.scala:14) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:471) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:444) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:545) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:540) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:350) at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:281) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:201) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.util.concurrent.TimeoutException: Timed out waiting for a cluster-wide sync to be released. (timeout = 240 seconds) at org.infinispan.remoting.transport.jgroups.JGroupsDistSync.blockUntilReleased(JGroupsDistSync.java:54) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:390) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:102) ... 61 more 2011-02-23 11:29:00,792 ERROR [org.infinispan.server.core.AbstractProtocolDecoder$] (HotRodServerWorker-1-1) Exception reported org.infinispan.CacheException: java.util.concurrent.TimeoutException: Timed out waiting for a cluster-wide sync to be released. (timeout = 240 seconds) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:115) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:126) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:231) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:218) at org.infinispan.remoting.rpc.RpcManagerImpl.broadcastRpcCommand(RpcManagerImpl.java:201) at org.infinispan.remoting.rpc.RpcManagerImpl.broadcastRpcCommand(RpcManagerImpl.java:194) at org.infinispan.interceptors.ReplicationInterceptor.handleCrudMethod(ReplicationInterceptor.java:115) at org.infinispan.interceptors.ReplicationInterceptor.visitReplaceCommand(ReplicationInterceptor.java:98) at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:63) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.LockingInterceptor.visitReplaceCommand(LockingInterceptor.java:338) at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:63) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) at org.infinispan.commands.AbstractVisitor.visitReplaceCommand(AbstractVisitor.java:66) at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:63) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:184) at org.infinispan.interceptors.TxInterceptor.visitReplaceCommand(TxInterceptor.java:142) at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:63) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) at org.infinispan.commands.AbstractVisitor.visitReplaceCommand(AbstractVisitor.java:66) at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:63) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:95) at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:62) at org.infinispan.commands.AbstractVisitor.visitReplaceCommand(AbstractVisitor.java:66) at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:63) at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:273) at org.infinispan.CacheDelegate.replace(CacheDelegate.java:482) at org.infinispan.CacheSupport.replace(CacheSupport.java:96) at org.infinispan.server.hotrod.HotRodEncoder.writeHashTopologyHeader(HotRodEncoder.scala:180) at org.infinispan.server.hotrod.HotRodEncoder.writeHeader(HotRodEncoder.scala:117) at org.infinispan.server.hotrod.HotRodEncoder.encode(HotRodEncoder.scala:32) at org.infinispan.server.core.transport.netty.EncoderAdapter.encode(EncoderAdapter.scala:17) at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:66) at org.jboss.netty.channel.Channels.write(Channels.java:611) at org.jboss.netty.channel.Channels.write(Channels.java:578) at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:259) at org.infinispan.server.core.transport.netty.ChannelAdapter.write(ChannelAdapter.scala:21) at org.infinispan.server.core.AbstractProtocolDecoder.writeResponse(AbstractProtocolDecoder.scala:87) at org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:51) at org.infinispan.server.core.transport.netty.DecoderAdapter.decode(DecoderAdapter.scala:18) at org.infinispan.server.core.transport.netty.DecoderAdapter.decode(DecoderAdapter.scala:14) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:471) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:444) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:350) at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:281) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:201) at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.util.concurrent.TimeoutException: Timed out waiting for a cluster-wide sync to be released. (timeout = 240 seconds) at org.infinispan.remoting.transport.jgroups.JGroupsDistSync.blockUntilReleased(JGroupsDistSync.java:54) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:390) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:102) ... 55 more 2011-02-23 11:29:40,065 INFO [org.infinispan.server.core.ShutdownHook] (ShutdownHookThread) Posting Shutdown Request to the server... 2011-02-23 11:29:40,082 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (HotRodServerWorker-1-1) Execution error: org.infinispan.CacheException: java.lang.RuntimeException: failed executing request org.jgroups.blocks.GroupRequest@13f99afreq_id=2, mode=GET_ALL, entries: localhost-43522: sender=localhost-43522, retval=null, received=false, suspected=false at org.infinispan.util.Util.rewrapAsCacheException(Util.java:468) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:118) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:394) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:102) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:126) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:231) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:218) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:214) at org.infinispan.interceptors.DistributionInterceptor.handleWriteCommand(DistributionInterceptor.java:386) at org.infinispan.interceptors.DistributionInterceptor.visitPutKeyValueCommand(DistributionInterceptor.java:191) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:80) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.LockingInterceptor.visitPutKeyValueCommand(LockingInterceptor.java:297) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:80) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:58) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:80) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:184) at org.infinispan.interceptors.TxInterceptor.visitPutKeyValueCommand(TxInterceptor.java:132) at org.infinispan.interceptors.DistTxInterceptor.visitPutKeyValueCommand(DistTxInterceptor.java:79) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:80) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:113) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:80) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:95) at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:62) at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:58) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:80) at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:273) at org.infinispan.CacheDelegate.put(CacheDelegate.java:450) at org.infinispan.server.core.AbstractProtocolDecoder.put(AbstractProtocolDecoder.scala:96) at org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:40) at org.infinispan.server.core.transport.netty.DecoderAdapter.decode(DecoderAdapter.scala:18) at org.infinispan.server.core.transport.netty.DecoderAdapter.decode(DecoderAdapter.scala:14) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:471) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:444) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:545) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:540) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:350) at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:281) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:201) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.RuntimeException: failed executing request org.jgroups.blocks.GroupRequest@13f99afreq_id=2, mode=GET_ALL, entries: localhost-43522: sender=localhost-43522, retval=null, received=false, suspected=false at org.jgroups.blocks.MessageDispatcher.cast(MessageDispatcher.java:532) at org.jgroups.blocks.MessageDispatcher.castMessage(MessageDispatcher.java:456) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$ReplicationTask.call(CommandAwareRpcDispatcher.java:232) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:116) ... 50 more Caused by: java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2105) at org.jgroups.blocks.Request.responsesComplete(Request.java:225) at org.jgroups.blocks.Request.execute(Request.java:118) at org.jgroups.blocks.MessageDispatcher.cast(MessageDispatcher.java:528) ... 53 more 2011-02-23 11:29:40,085 ERROR [org.infinispan.server.core.AbstractProtocolDecoder$] (HotRodServerWorker-1-1) Exception reported org.infinispan.server.hotrod.HotRodException: org.infinispan.CacheException: java.lang.RuntimeException: failed executing request org.jgroups.blocks.GroupRequest@13f99afreq_id=2, mode=GET_ALL, entries: localhost-43522: sender=localhost-43522, retval=null, received=false, suspected=false at org.infinispan.server.hotrod.HotRodDecoder.createServerException(HotRodDecoder.scala:144) at org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:55) at org.infinispan.server.core.transport.netty.DecoderAdapter.decode(DecoderAdapter.scala:18) at org.infinispan.server.core.transport.netty.DecoderAdapter.decode(DecoderAdapter.scala:14) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:471) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:444) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:350) at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:281) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:201) at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: org.infinispan.CacheException: java.lang.RuntimeException: failed executing request org.jgroups.blocks.GroupRequest@13f99afreq_id=2, mode=GET_ALL, entries: localhost-43522: sender=localhost-43522, retval=null, received=false, suspected=false at org.infinispan.util.Util.rewrapAsCacheException(Util.java:468) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:118) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:394) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:102) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:126) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:231) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:218) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:214) at org.infinispan.interceptors.DistributionInterceptor.handleWriteCommand(DistributionInterceptor.java:386) at org.infinispan.interceptors.DistributionInterceptor.visitPutKeyValueCommand(DistributionInterceptor.java:191) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:80) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.LockingInterceptor.visitPutKeyValueCommand(LockingInterceptor.java:297) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:80) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:58) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:80) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:184) at org.infinispan.interceptors.TxInterceptor.visitPutKeyValueCommand(TxInterceptor.java:132) at org.infinispan.interceptors.DistTxInterceptor.visitPutKeyValueCommand(DistTxInterceptor.java:79) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:80) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:113) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:80) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:95) at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:62) at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:58) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:80) at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:273) at org.infinispan.CacheDelegate.put(CacheDelegate.java:450) at org.infinispan.server.core.AbstractProtocolDecoder.put(AbstractProtocolDecoder.scala:96) at org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:40) ... 13 more Caused by: java.lang.RuntimeException: failed executing request org.jgroups.blocks.GroupRequest@13f99afreq_id=2, mode=GET_ALL, entries: localhost-43522: sender=localhost-43522, retval=null, received=false, suspected=false at org.jgroups.blocks.MessageDispatcher.cast(MessageDispatcher.java:532) at org.jgroups.blocks.MessageDispatcher.castMessage(MessageDispatcher.java:456) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$ReplicationTask.call(CommandAwareRpcDispatcher.java:232) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:116) ... 46 more Caused by: java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2105) at org.jgroups.blocks.Request.responsesComplete(Request.java:225) at org.jgroups.blocks.Request.execute(Request.java:118) at org.jgroups.blocks.MessageDispatcher.cast(MessageDispatcher.java:528) ... 49 more 2011-02-23 11:29:40,191 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (HotRodServerWorker-1-1) Execution error: org.infinispan.CacheException: java.lang.RuntimeException: failed executing request org.jgroups.blocks.GroupRequest@1c81ac0req_id=3, mode=GET_ALL, entries: localhost-43522: sender=localhost-43522, retval=null, received=false, suspected=false at org.infinispan.util.Util.rewrapAsCacheException(Util.java:468) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:118) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:394) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:102) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:126) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:231) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:218) at org.infinispan.remoting.rpc.RpcManagerImpl.broadcastRpcCommand(RpcManagerImpl.java:201) at org.infinispan.remoting.rpc.RpcManagerImpl.broadcastRpcCommand(RpcManagerImpl.java:194) at org.infinispan.interceptors.ReplicationInterceptor.handleCrudMethod(ReplicationInterceptor.java:115) at org.infinispan.interceptors.ReplicationInterceptor.visitReplaceCommand(ReplicationInterceptor.java:98) at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:63) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.LockingInterceptor.visitReplaceCommand(LockingInterceptor.java:338) at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:63) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) at org.infinispan.commands.AbstractVisitor.visitReplaceCommand(AbstractVisitor.java:66) at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:63) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:184) at org.infinispan.interceptors.TxInterceptor.visitReplaceCommand(TxInterceptor.java:142) at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:63) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) at org.infinispan.commands.AbstractVisitor.visitReplaceCommand(AbstractVisitor.java:66) at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:63) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:95) at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:62) at org.infinispan.commands.AbstractVisitor.visitReplaceCommand(AbstractVisitor.java:66) at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:63) at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:273) at org.infinispan.CacheDelegate.replace(CacheDelegate.java:482) at org.infinispan.CacheSupport.replace(CacheSupport.java:96) at org.infinispan.server.hotrod.HotRodEncoder.writeHashTopologyHeader(HotRodEncoder.scala:180) at org.infinispan.server.hotrod.HotRodEncoder.writeHeader(HotRodEncoder.scala:117) at org.infinispan.server.hotrod.HotRodEncoder.encode(HotRodEncoder.scala:32) at org.infinispan.server.core.transport.netty.EncoderAdapter.encode(EncoderAdapter.scala:17) at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:66) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:568) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:563) at org.jboss.netty.channel.Channels.write(Channels.java:611) at org.jboss.netty.channel.Channels.write(Channels.java:578) at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:259) at org.infinispan.server.core.transport.netty.ChannelAdapter.write(ChannelAdapter.scala:21) at org.infinispan.server.core.AbstractProtocolDecoder.exceptionCaught(AbstractProtocolDecoder.scala:168) at org.infinispan.server.core.transport.netty.DecoderAdapter.exceptionCaught(DecoderAdapter.scala:22) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:545) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:540) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:432) at org.jboss.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:52) at org.jboss.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:622) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:547) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:540) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:350) at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:281) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:201) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.RuntimeException: failed executing request org.jgroups.blocks.GroupRequest@1c81ac0req_id=3, mode=GET_ALL, entries: localhost-43522: sender=localhost-43522, retval=null, received=false, suspected=false at org.jgroups.blocks.MessageDispatcher.cast(MessageDispatcher.java:532) at org.jgroups.blocks.MessageDispatcher.castMessage(MessageDispatcher.java:456) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$ReplicationTask.call(CommandAwareRpcDispatcher.java:232) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:116) ... 64 more Caused by: java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2105) at org.jgroups.blocks.Request.responsesComplete(Request.java:225) at org.jgroups.blocks.Request.execute(Request.java:118) at org.jgroups.blocks.MessageDispatcher.cast(MessageDispatcher.java:528) ... 67 more 2011-02-23 11:29:40,193 ERROR [org.infinispan.server.core.AbstractProtocolDecoder$] (HotRodServerWorker-1-1) Exception reported org.infinispan.CacheException: java.lang.RuntimeException: failed executing request org.jgroups.blocks.GroupRequest@1c81ac0req_id=3, mode=GET_ALL, entries: localhost-43522: sender=localhost-43522, retval=null, received=false, suspected=false at org.infinispan.util.Util.rewrapAsCacheException(Util.java:468) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:118) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:394) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:102) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:126) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:231) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:218) at org.infinispan.remoting.rpc.RpcManagerImpl.broadcastRpcCommand(RpcManagerImpl.java:201) at org.infinispan.remoting.rpc.RpcManagerImpl.broadcastRpcCommand(RpcManagerImpl.java:194) at org.infinispan.interceptors.ReplicationInterceptor.handleCrudMethod(ReplicationInterceptor.java:115) at org.infinispan.interceptors.ReplicationInterceptor.visitReplaceCommand(ReplicationInterceptor.java:98) at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:63) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.LockingInterceptor.visitReplaceCommand(LockingInterceptor.java:338) at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:63) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) at org.infinispan.commands.AbstractVisitor.visitReplaceCommand(AbstractVisitor.java:66) at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:63) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:184) at org.infinispan.interceptors.TxInterceptor.visitReplaceCommand(TxInterceptor.java:142) at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:63) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) at org.infinispan.commands.AbstractVisitor.visitReplaceCommand(AbstractVisitor.java:66) at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:63) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:95) at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:62) at org.infinispan.commands.AbstractVisitor.visitReplaceCommand(AbstractVisitor.java:66) at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:63) at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:273) at org.infinispan.CacheDelegate.replace(CacheDelegate.java:482) at org.infinispan.CacheSupport.replace(CacheSupport.java:96) at org.infinispan.server.hotrod.HotRodEncoder.writeHashTopologyHeader(HotRodEncoder.scala:180) at org.infinispan.server.hotrod.HotRodEncoder.writeHeader(HotRodEncoder.scala:117) at org.infinispan.server.hotrod.HotRodEncoder.encode(HotRodEncoder.scala:32) at org.infinispan.server.core.transport.netty.EncoderAdapter.encode(EncoderAdapter.scala:17) at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:66) at org.jboss.netty.channel.Channels.write(Channels.java:611) at org.jboss.netty.channel.Channels.write(Channels.java:578) at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:259) at org.infinispan.server.core.transport.netty.ChannelAdapter.write(ChannelAdapter.scala:21) at org.infinispan.server.core.AbstractProtocolDecoder.exceptionCaught(AbstractProtocolDecoder.scala:168) at org.infinispan.server.core.transport.netty.DecoderAdapter.exceptionCaught(DecoderAdapter.scala:22) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:432) at org.jboss.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:52) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:350) at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:281) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:201) at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.RuntimeException: failed executing request org.jgroups.blocks.GroupRequest@1c81ac0req_id=3, mode=GET_ALL, entries: localhost-43522: sender=localhost-43522, retval=null, received=false, suspected=false at org.jgroups.blocks.MessageDispatcher.cast(MessageDispatcher.java:532) at org.jgroups.blocks.MessageDispatcher.castMessage(MessageDispatcher.java:456) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$ReplicationTask.call(CommandAwareRpcDispatcher.java:232) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:116) ... 55 more Caused by: java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2105) at org.jgroups.blocks.Request.responsesComplete(Request.java:225) at org.jgroups.blocks.Request.execute(Request.java:118) at org.jgroups.blocks.MessageDispatcher.cast(MessageDispatcher.java:528) ... 58 more 2011-02-23 11:29:40,198 ERROR [org.infinispan.server.core.AbstractProtocolDecoder$] (HotRodServerWorker-1-1) Exception reported java.nio.channels.ClosedChannelException at org.jboss.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:637) at org.jboss.netty.channel.socket.nio.NioWorker.writeFromUserCode(NioWorker.java:370) at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:137) at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:76) at org.jboss.netty.channel.Channels.write(Channels.java:632) at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:70) at org.jboss.netty.channel.Channels.write(Channels.java:611) at org.jboss.netty.channel.Channels.write(Channels.java:578) at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:259) at org.infinispan.server.core.transport.netty.ChannelAdapter.write(ChannelAdapter.scala:21) at org.infinispan.server.core.AbstractProtocolDecoder.exceptionCaught(AbstractProtocolDecoder.scala:168) at org.infinispan.server.core.transport.netty.DecoderAdapter.exceptionCaught(DecoderAdapter.scala:22) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:432) at org.jboss.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:52) at org.jboss.netty.channel.Channels.write(Channels.java:611) at org.jboss.netty.channel.Channels.write(Channels.java:578) at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:259) at org.infinispan.server.core.transport.netty.ChannelAdapter.write(ChannelAdapter.scala:21) at org.infinispan.server.core.AbstractProtocolDecoder.exceptionCaught(AbstractProtocolDecoder.scala:168) at org.infinispan.server.core.transport.netty.DecoderAdapter.exceptionCaught(DecoderAdapter.scala:22) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:432) at org.jboss.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:52) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:350) at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:281) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:201) at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2011-02-23 11:29:44,590 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-5,localhost-42248) Received new cluster view: [localhost-42248|2] [localhost-42248] 2011-02-23 11:29:44,592 INFO [org.infinispan.distribution.DistributionManagerImpl] (Incoming-5,localhost-42248) Detected a view change. Member list changed from [localhost-42248, localhost-43522] to [localhost-42248] 2011-02-23 11:29:44,592 INFO [org.infinispan.distribution.DistributionManagerImpl] (Incoming-5,localhost-42248) This is a LEAVE event! Node localhost-43522 has just left 2011-02-23 11:29:44,593 INFO [org.infinispan.distribution.DistributionManagerImpl] (Incoming-5,localhost-42248) I localhost-42248 am participating in rehash, state providers [localhost-42248, localhost-42248], state receivers [localhost-42248, localhost-42248] 2011-02-23 11:29:44,603 INFO [org.infinispan.distribution.InvertedLeaveTask] (Rehasher-localhost-42248) Completed leave rehash on node localhost-42248 in 7 milliseconds - leavers now are []