1. michal-linhard-37465 (test1) tries to retrieve state from michal-linhard-12702 (test3) and sends a request to open the flush wait gate (StateTransferControlCommand).

15:39:22,611 INFO  [org.infinispan.remoting.rpc.RpcManagerImpl] (MSC service thread 1-4) ISPN000074: Trying to fetch state from michal-linhard-12702
15:39:22,612 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) dests=[michal-linhard-12702], command=StateTransferControlCommand{enabled=true}, mode=SYNCHRONOUS, timeout=480000
15:39:22,612 TRACE [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (MSC service thread 1-4) Replication task sending StateTransferControlCommand{enabled=true} to addresses [michal-linhard-12702]
15:39:22,618 TRACE [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (MSC service thread 1-4) Responses: [sender=michal-linhard-12702, retval=null, received=true, suspected=false]

2. michal-linhard-61619 (test4) tries to retrieve state from michal-linhard-12702 (test3) and sends a request to open the flush wait gate (StateTransferControlCommand).

15:39:22,640 INFO  [org.infinispan.remoting.rpc.RpcManagerImpl] (MSC service thread 1-1) ISPN000074: Trying to fetch state from michal-linhard-12702
15:39:22,640 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) dests=[michal-linhard-12702], command=StateTransferControlCommand{enabled=true}, mode=SYNCHRONOUS, timeout=480000
15:39:22,640 TRACE [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (MSC service thread 1-1) Replication task sending StateTransferControlCommand{enabled=true} to addresses [michal-linhard-12702]
15:39:22,644 TRACE [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (MSC service thread 1-1) Responses: [sender=michal-linhard-12702, retval=null, received=true, suspected=false]

3. michal-linhard-12702 (test3) receives both requests, and it tells michal-linhard-37465 (test1) that it can provide state, and it tells michal-linhard-61619 (test4) that it cannot.

15:39:22,681 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (STREAMING_STATE_TRANSFER-sender-1,default,michal-linhard-12702) Received request to generate state for cache named '___hotRodTopologyCache'.  Attempting to generate state.
15:39:22,682 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (STREAMING_STATE_TRANSFER-sender-1,default,michal-linhard-12702) Generating state.  Can provide? true
15:39:22,687 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (STREAMING_STATE_TRANSFER-sender-1,default,michal-linhard-12702) Writing 1 StoredEntries to stream
15:39:22,688 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (STREAMING_STATE_TRANSFER-sender-2,default,michal-linhard-12702) Received request to generate state for cache named '___hotRodTopologyCache'.  Attempting to generate state.
15:39:22,688 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (STREAMING_STATE_TRANSFER-sender-2,default,michal-linhard-12702) Generating state.  Can provide? false
15:39:22,688 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (STREAMING_STATE_TRANSFER-sender-2,default,michal-linhard-12702) Not providing state!

3. As a result, michal-linhard-61619 (test4) fails consuming the state because the receiver was in the middle of another request:

15:39:22,699 ERROR [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-5-thread-44) ISPN000096: Caught while requesting or applying state: org.infinispan.statetransfer.StateTransferException: Provider cannot provide state!
	at org.infinispan.statetransfer.StateTransferManagerImpl.applyState(StateTransferManagerImpl.java:286) [infinispan-core-5.0.0-SNAPSHOT.jar:5.0.0-SNAPSHOT]
	at org.infinispan.remoting.InboundInvocationHandlerImpl.applyState(InboundInvocationHandlerImpl.java:230) [infinispan-core-5.0.0-SNAPSHOT.jar:5.0.0-SNAPSHOT]
 
4. More importantly, this leads to michal-linhard-61619 (test4) closing the flush wait gate in michal-linhard-12702 (test3)

15:39:22,700 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) dests=[michal-linhard-12702], command=StateTransferControlCommand{enabled=false}, mode=SYNCHRONOUS, timeout=480000
15:39:22,701 TRACE [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (MSC service thread 1-1) Replication task sending StateTransferControlCommand{enabled=false} to addresses [michal-linhard-12702]
15:39:22,702 TRACE [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (MSC service thread 1-1) Responses: [sender=michal-linhard-12702, retval=null, received=true, suspected=false]

5. And this results in the michal-linhard-12702 (test3) failing to find the flush wait gate open when trying to reply to michal-linhard-37465 (test1):

15:40:22,698 ERROR [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (STREAMING_STATE_TRANSFER-sender-1,default,michal-linhard-12702) ISPN000095: Caught while responding to state transfer request: org.infinispan.statetransfer.StateTransferException: java.util.concurrent.TimeoutException: Timed out waiting for a cluster-wide sync to be acquired. (timeout = 60 seconds)
	at org.infinispan.statetransfer.StateTransferManagerImpl.generateState(StateTransferManagerImpl.java:162) [infinispan-core-5.0.0-SNAPSHOT.jar:5.0.0-SNAPSHOT]
	at org.infinispan.remoting.InboundInvocationHandlerImpl.generateState(InboundInvocationHandlerImpl.java:248) [infinispan-core-5.0.0-SNAPSHOT.jar:5.0.0-SNAPSHOT]
	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.getState(JGroupsTransport.java:590) [infinispan-core-5.0.0-SNAPSHOT.jar:5.0.0-SNAPSHOT]
	at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUpEvent(MessageDispatcher.java:690) [jgroups-2.12.1.Final.jar:]
	at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:771) [jgroups-2.12.1.Final.jar:]
	at org.jgroups.JChannel.up(JChannel.java:1484) [jgroups-2.12.1.Final.jar:]
	at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1074) [jgroups-2.12.1.Final.jar:]
	at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:477) [jgroups-2.12.1.Final.jar:]
	at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER$StateProviderHandler.process(STREAMING_STATE_TRANSFER.java:651) [jgroups-2.12.1.Final.jar:]
	at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER$StateProviderThreadSpawner$1.run(STREAMING_STATE_TRANSFER.java:580) [jgroups-2.12.1.Final.jar:]
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [:1.6.0_25]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [:1.6.0_25]
	at java.lang.Thread.run(Thread.java:662) [:1.6.0_25]
Caused by: java.util.concurrent.TimeoutException: Timed out waiting for a cluster-wide sync to be acquired. (timeout = 60 seconds)
	at org.infinispan.remoting.transport.jgroups.JGroupsDistSync.blockUntilAcquired(JGroupsDistSync.java:62) [infinispan-core-5.0.0-SNAPSHOT.jar:5.0.0-SNAPSHOT]
	at org.infinispan.statetransfer.StateTransferManagerImpl.generateTransactionLog(StateTransferManagerImpl.java:196) [infinispan-core-5.0.0-SNAPSHOT.jar:5.0.0-SNAPSHOT]
	at org.infinispan.statetransfer.StateTransferManagerImpl.generateState(StateTransferManagerImpl.java:152) [infinispan-core-5.0.0-SNAPSHOT.jar:5.0.0-SNAPSHOT]
	... 12 more