2011-06-09 13:43:44,895 DEBUG [JBossMarshaller] (main) Using JBoss Marshalling 2011-06-09 13:43:44,952 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.transaction.xa.recovery.InDoubtTxInfoImpl$Externalizer for class org.infinispan.transaction.xa.recovery.InDoubtTxInfoImpl with id 70 and reader index 70 2011-06-09 13:43:44,953 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.distribution.RemoteTransactionLogDetails$Externalizer for class org.infinispan.distribution.RemoteTransactionLogDetails with id 63 and reader index 63 2011-06-09 13:43:44,953 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.transaction.TransactionLog$LogEntry$Externalizer for class org.infinispan.transaction.TransactionLog$LogEntry with id 41 and reader index 41 2011-06-09 13:43:44,954 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.util.ByteArrayKey$Externalizer for class org.infinispan.util.ByteArrayKey with id 57 and reader index 57 2011-06-09 13:43:44,955 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.remoting.responses.UnsuccessfulResponse$Externalizer for class org.infinispan.remoting.responses.UnsuccessfulResponse with id 8 and reader index 8 2011-06-09 13:43:44,956 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.marshall.exts.LinkedListExternalizer for class java.util.LinkedList with id 1 and reader index 1 2011-06-09 13:43:44,957 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.atomic.AtomicHashMapDelta$Externalizer for class org.infinispan.atomic.AtomicHashMapDelta with id 46 and reader index 46 2011-06-09 13:43:44,958 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.util.Immutables$ImmutableMapWrapperExternalizer for class org.infinispan.util.Immutables$ImmutableMapWrapper with id 19 and reader index 19 2011-06-09 13:43:44,959 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.atomic.ClearOperation$Externalizer for class org.infinispan.atomic.ClearOperation with id 49 and reader index 49 2011-06-09 13:43:44,961 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.container.entries.TransientCacheValue$Externalizer for class org.infinispan.container.entries.TransientCacheValue with id 16 and reader index 16 2011-06-09 13:43:44,961 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.container.entries.TransientCacheEntry$Externalizer for class org.infinispan.container.entries.TransientCacheEntry with id 12 and reader index 12 2011-06-09 13:43:44,962 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.container.entries.MortalCacheEntry$Externalizer for class org.infinispan.container.entries.MortalCacheEntry with id 11 and reader index 11 2011-06-09 13:43:44,963 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.atomic.PutOperation$Externalizer for class org.infinispan.atomic.PutOperation with id 47 and reader index 47 2011-06-09 13:43:44,963 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.marshall.exts.MapExternalizer for class java.util.TreeMap with id 2 and reader index 2 2011-06-09 13:43:44,963 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.marshall.exts.MapExternalizer for class java.util.HashMap with id 2 and reader index 2 2011-06-09 13:43:44,964 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.marshall.exts.MapExternalizer for class org.infinispan.util.FastCopyHashMap with id 2 and reader index 2 2011-06-09 13:43:44,965 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.remoting.responses.UnsureResponse$Externalizer for class org.infinispan.remoting.responses.UnsureResponse with id 54 and reader index 54 2011-06-09 13:43:44,966 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.distribution.ch.TopologyAwareConsistentHash$Externalizer for class org.infinispan.distribution.ch.TopologyAwareConsistentHash with id 61 and reader index 61 2011-06-09 13:43:44,966 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.transaction.xa.GlobalTransaction$Externalizer for class org.infinispan.transaction.xa.GlobalTransaction with id 38 and reader index 38 2011-06-09 13:43:44,968 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.atomic.AtomicHashMap$Externalizer for class org.infinispan.atomic.AtomicHashMap with id 20 and reader index 20 2011-06-09 13:43:44,968 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.transaction.xa.recovery.RecoveryAwareGlobalTransaction$Externalizer for class org.infinispan.transaction.xa.recovery.RecoveryAwareGlobalTransaction with id 68 and reader index 68 2011-06-09 13:43:44,969 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.container.entries.ImmortalCacheEntry$Externalizer for class org.infinispan.container.entries.ImmortalCacheEntry with id 10 and reader index 10 2011-06-09 13:43:44,980 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.marshall.exts.SingletonListExternalizer for class java.util.Collections$SingletonList with id 4 and reader index 4 2011-06-09 13:43:44,982 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.remoting.responses.ExtendedResponse$Externalizer for class org.infinispan.remoting.responses.ExtendedResponse with id 6 and reader index 6 2011-06-09 13:43:44,984 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.marshall.MarshalledValue$Externalizer for class org.infinispan.marshall.MarshalledValue with id 40 and reader index 40 2011-06-09 13:43:44,989 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.container.entries.TransientMortalCacheEntry$Externalizer for class org.infinispan.container.entries.TransientMortalCacheEntry with id 13 and reader index 13 2011-06-09 13:43:44,990 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.remoting.transport.jgroups.JGroupsAddress$Externalizer for class org.infinispan.remoting.transport.jgroups.JGroupsAddress with id 39 and reader index 39 2011-06-09 13:43:44,990 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.transaction.xa.recovery.RecoveryAwareDldGlobalTransaction$Externalizer for class org.infinispan.transaction.xa.recovery.RecoveryAwareDldGlobalTransaction with id 67 and reader index 67 2011-06-09 13:43:44,991 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.marshall.exts.ReplicableCommandExternalizer for class org.infinispan.commands.write.InvalidateL1Command with id 62 and reader index 62 2011-06-09 13:43:44,992 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.marshall.exts.ReplicableCommandExternalizer for class org.infinispan.commands.tx.PrepareCommand with id 62 and reader index 62 2011-06-09 13:43:45,004 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.marshall.exts.ReplicableCommandExternalizer for class org.infinispan.commands.remote.ClusteredGetCommand with id 62 and reader index 62 2011-06-09 13:43:45,004 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.marshall.exts.ReplicableCommandExternalizer for class org.infinispan.commands.remote.recovery.CompleteTransactionCommand with id 62 and reader index 62 2011-06-09 13:43:45,004 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.marshall.exts.ReplicableCommandExternalizer for class org.infinispan.commands.RemoveCacheCommand with id 62 and reader index 62 2011-06-09 13:43:45,005 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.marshall.exts.ReplicableCommandExternalizer for class org.infinispan.commands.remote.recovery.RemoveRecoveryInfoCommand with id 62 and reader index 62 2011-06-09 13:43:45,005 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.marshall.exts.ReplicableCommandExternalizer for class org.infinispan.commands.tx.RollbackCommand with id 62 and reader index 62 2011-06-09 13:43:45,005 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.marshall.exts.ReplicableCommandExternalizer for class org.infinispan.commands.remote.MultipleRpcCommand with id 62 and reader index 62 2011-06-09 13:43:45,005 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.marshall.exts.ReplicableCommandExternalizer for class org.infinispan.commands.read.MapReduceCommand with id 62 and reader index 62 2011-06-09 13:43:45,006 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.marshall.exts.ReplicableCommandExternalizer for class org.infinispan.commands.write.PutKeyValueCommand with id 62 and reader index 62 2011-06-09 13:43:45,006 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.marshall.exts.ReplicableCommandExternalizer for class org.infinispan.commands.tx.CommitCommand with id 62 and reader index 62 2011-06-09 13:43:45,006 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.marshall.exts.ReplicableCommandExternalizer for class org.infinispan.commands.write.RemoveCommand with id 62 and reader index 62 2011-06-09 13:43:45,007 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.marshall.exts.ReplicableCommandExternalizer for class org.infinispan.commands.control.StateTransferControlCommand with id 62 and reader index 62 2011-06-09 13:43:45,008 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.marshall.exts.ReplicableCommandExternalizer for class org.infinispan.commands.read.GetKeyValueCommand with id 62 and reader index 62 2011-06-09 13:43:45,008 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.marshall.exts.ReplicableCommandExternalizer for class org.infinispan.commands.write.ClearCommand with id 62 and reader index 62 2011-06-09 13:43:45,008 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.marshall.exts.ReplicableCommandExternalizer for class org.infinispan.commands.write.ReplaceCommand with id 62 and reader index 62 2011-06-09 13:43:45,008 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.marshall.exts.ReplicableCommandExternalizer for class org.infinispan.commands.remote.SingleRpcCommand with id 62 and reader index 62 2011-06-09 13:43:45,009 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.marshall.exts.ReplicableCommandExternalizer for class org.infinispan.commands.write.InvalidateCommand with id 62 and reader index 62 2011-06-09 13:43:45,009 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.marshall.exts.ReplicableCommandExternalizer for class org.infinispan.commands.control.LockControlCommand with id 62 and reader index 62 2011-06-09 13:43:45,023 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.marshall.exts.ReplicableCommandExternalizer for class org.infinispan.commands.remote.recovery.GetInDoubtTransactionsCommand with id 62 and reader index 62 2011-06-09 13:43:45,024 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.marshall.exts.ReplicableCommandExternalizer for class org.infinispan.commands.remote.recovery.GetInDoubtTxInfoCommand with id 62 and reader index 62 2011-06-09 13:43:45,024 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.marshall.exts.ReplicableCommandExternalizer for class org.infinispan.commands.write.PutMapCommand with id 62 and reader index 62 2011-06-09 13:43:45,024 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.marshall.exts.ReplicableCommandExternalizer for class org.infinispan.commands.control.RehashControlCommand with id 62 and reader index 62 2011-06-09 13:43:45,024 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.marshall.exts.ReplicableCommandExternalizer for class org.infinispan.commands.write.EvictCommand with id 62 and reader index 62 2011-06-09 13:43:45,025 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.marshall.exts.ReplicableCommandExternalizer for class org.infinispan.commands.read.DistributedExecuteCommand with id 62 and reader index 62 2011-06-09 13:43:45,025 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.marshall.exts.ArrayListExternalizer for class java.util.ArrayList with id 0 and reader index 0 2011-06-09 13:43:45,026 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.container.entries.TransientMortalCacheValue$Externalizer for class org.infinispan.container.entries.TransientMortalCacheValue with id 17 and reader index 17 2011-06-09 13:43:45,026 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.container.entries.MortalCacheValue$Externalizer for class org.infinispan.container.entries.MortalCacheValue with id 15 and reader index 15 2011-06-09 13:43:45,027 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.remoting.transport.jgroups.JGroupsTopologyAwareAddress$Externalizer for class org.infinispan.remoting.transport.jgroups.JGroupsTopologyAwareAddress with id 60 and reader index 60 2011-06-09 13:43:45,028 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.remoting.responses.RequestIgnoredResponse$Externalizer for class org.infinispan.remoting.responses.RequestIgnoredResponse with id 9 and reader index 9 2011-06-09 13:43:45,028 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.marshall.exts.SetExternalizer for class java.util.TreeSet with id 3 and reader index 3 2011-06-09 13:43:45,029 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.marshall.exts.SetExternalizer for class java.util.HashSet with id 3 and reader index 3 2011-06-09 13:43:45,029 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.atomic.RemoveOperation$Externalizer for class org.infinispan.atomic.RemoveOperation with id 48 and reader index 48 2011-06-09 13:43:45,030 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.remoting.responses.ExceptionResponse$Externalizer for class org.infinispan.remoting.responses.ExceptionResponse with id 7 and reader index 7 2011-06-09 13:43:45,031 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.remoting.responses.SuccessfulResponse$Externalizer for class org.infinispan.remoting.responses.SuccessfulResponse with id 5 and reader index 5 2011-06-09 13:43:45,031 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.transaction.xa.recovery.SerializableXid$XidExternalizer for class org.infinispan.transaction.xa.recovery.SerializableXid with id 66 and reader index 66 2011-06-09 13:43:45,032 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.loaders.bucket.Bucket$Externalizer for class org.infinispan.loaders.bucket.Bucket with id 42 and reader index 42 2011-06-09 13:43:45,033 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.distribution.ch.UnionConsistentHash$Externalizer for class org.infinispan.distribution.ch.UnionConsistentHash with id 52 and reader index 52 2011-06-09 13:43:45,033 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.container.entries.ImmortalCacheValue$Externalizer for class org.infinispan.container.entries.ImmortalCacheValue with id 14 and reader index 14 2011-06-09 13:43:45,034 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.distribution.ch.VirtualAddress$Externalizer for class org.infinispan.distribution.ch.VirtualAddress with id 69 and reader index 69 2011-06-09 13:43:45,034 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.distribution.ch.DefaultConsistentHash$Externalizer for class org.infinispan.distribution.ch.DefaultConsistentHash with id 51 and reader index 51 2011-06-09 13:43:45,034 TRACE [ExternalizerTable] (main) Loaded externalizer org.infinispan.transaction.xa.DldGlobalTransaction$Externalizer for class org.infinispan.transaction.xa.DldGlobalTransaction with id 43 and reader index 43 2011-06-09 13:43:45,034 TRACE [ExternalizerTable] (main) Loading user defined externalizers 2011-06-09 13:43:45,035 TRACE [ExternalizerTable] (main) Constant object table was started and contains these externalizer readers: {0=org.infinispan.marshall.exts.ArrayListExternalizer, 68=org.infinispan.transaction.xa.recovery.RecoveryAwareGlobalTransaction$Externalizer, 69=org.infinispan.distribution.ch.VirtualAddress$Externalizer, 1=org.infinispan.marshall.exts.LinkedListExternalizer, 70=org.infinispan.transaction.xa.recovery.InDoubtTxInfoImpl$Externalizer, 2=org.infinispan.marshall.exts.MapExternalizer, 3=org.infinispan.marshall.exts.SetExternalizer, 4=org.infinispan.marshall.exts.SingletonListExternalizer, 5=org.infinispan.remoting.responses.SuccessfulResponse$Externalizer, 66=org.infinispan.transaction.xa.recovery.SerializableXid$XidExternalizer, 6=org.infinispan.remoting.responses.ExtendedResponse$Externalizer, 7=org.infinispan.remoting.responses.ExceptionResponse$Externalizer, 67=org.infinispan.transaction.xa.recovery.RecoveryAwareDldGlobalTransaction$Externalizer, 8=org.infinispan.remoting.responses.UnsuccessfulResponse$Externalizer, 9=org.infinispan.remoting.responses.RequestIgnoredResponse$Externalizer, 10=org.infinispan.container.entries.ImmortalCacheEntry$Externalizer, 11=org.infinispan.container.entries.MortalCacheEntry$Externalizer, 12=org.infinispan.container.entries.TransientCacheEntry$Externalizer, 13=org.infinispan.container.entries.TransientMortalCacheEntry$Externalizer, 14=org.infinispan.container.entries.ImmortalCacheValue$Externalizer, 15=org.infinispan.container.entries.MortalCacheValue$Externalizer, 17=org.infinispan.container.entries.TransientMortalCacheValue$Externalizer, 16=org.infinispan.container.entries.TransientCacheValue$Externalizer, 19=org.infinispan.util.Immutables$ImmutableMapWrapperExternalizer, 20=org.infinispan.atomic.AtomicHashMap$Externalizer, 38=org.infinispan.transaction.xa.GlobalTransaction$Externalizer, 39=org.infinispan.remoting.transport.jgroups.JGroupsAddress$Externalizer, 42=org.infinispan.loaders.bucket.Bucket$Externalizer, 43=org.infinispan.transaction.xa.DldGlobalTransaction$Externalizer, 40=org.infinispan.marshall.MarshalledValue$Externalizer, 41=org.infinispan.transaction.TransactionLog$LogEntry$Externalizer, 46=org.infinispan.atomic.AtomicHashMapDelta$Externalizer, 47=org.infinispan.atomic.PutOperation$Externalizer, 51=org.infinispan.distribution.ch.DefaultConsistentHash$Externalizer, 49=org.infinispan.atomic.ClearOperation$Externalizer, 48=org.infinispan.atomic.RemoveOperation$Externalizer, 54=org.infinispan.remoting.responses.UnsureResponse$Externalizer, 52=org.infinispan.distribution.ch.UnionConsistentHash$Externalizer, 57=org.infinispan.util.ByteArrayKey$Externalizer, 63=org.infinispan.distribution.RemoteTransactionLogDetails$Externalizer, 62=org.infinispan.marshall.exts.ReplicableCommandExternalizer, 61=org.infinispan.distribution.ch.TopologyAwareConsistentHash$Externalizer, 60=org.infinispan.remoting.transport.jgroups.JGroupsTopologyAwareAddress$Externalizer} 2011-06-09 13:43:45,036 TRACE [ExternalizerTable] (main) The externalizer writers collection contains: {class org.infinispan.distribution.ch.VirtualAddress=org.infinispan.distribution.ch.VirtualAddress$Externalizer, class org.infinispan.commands.control.RehashControlCommand=org.infinispan.marshall.exts.ReplicableCommandExternalizer, class org.infinispan.commands.write.PutMapCommand=org.infinispan.marshall.exts.ReplicableCommandExternalizer, class org.infinispan.commands.remote.recovery.GetInDoubtTransactionsCommand=org.infinispan.marshall.exts.ReplicableCommandExternalizer, class org.infinispan.commands.control.LockControlCommand=org.infinispan.marshall.exts.ReplicableCommandExternalizer, class org.infinispan.util.FastCopyHashMap=org.infinispan.marshall.exts.MapExternalizer, class org.infinispan.loaders.bucket.Bucket=org.infinispan.loaders.bucket.Bucket$Externalizer, class org.infinispan.marshall.MarshalledValue=org.infinispan.marshall.MarshalledValue$Externalizer, class org.infinispan.commands.write.ReplaceCommand=org.infinispan.marshall.exts.ReplicableCommandExternalizer, class org.infinispan.commands.remote.SingleRpcCommand=org.infinispan.marshall.exts.ReplicableCommandExternalizer, class org.infinispan.commands.write.ClearCommand=org.infinispan.marshall.exts.ReplicableCommandExternalizer, class org.infinispan.commands.read.GetKeyValueCommand=org.infinispan.marshall.exts.ReplicableCommandExternalizer, class org.infinispan.commands.control.StateTransferControlCommand=org.infinispan.marshall.exts.ReplicableCommandExternalizer, class java.util.Collections$SingletonList=org.infinispan.marshall.exts.SingletonListExternalizer, class org.infinispan.transaction.xa.GlobalTransaction=org.infinispan.transaction.xa.GlobalTransaction$Externalizer, class java.util.ArrayList=org.infinispan.marshall.exts.ArrayListExternalizer, class org.infinispan.container.entries.ImmortalCacheEntry=org.infinispan.container.entries.ImmortalCacheEntry$Externalizer, class java.util.TreeMap=org.infinispan.marshall.exts.MapExternalizer, class org.infinispan.atomic.AtomicHashMapDelta=org.infinispan.atomic.AtomicHashMapDelta$Externalizer, class org.infinispan.atomic.PutOperation=org.infinispan.atomic.PutOperation$Externalizer, class org.infinispan.commands.read.MapReduceCommand=org.infinispan.marshall.exts.ReplicableCommandExternalizer, class org.infinispan.commands.tx.RollbackCommand=org.infinispan.marshall.exts.ReplicableCommandExternalizer, class java.util.LinkedList=org.infinispan.marshall.exts.LinkedListExternalizer, class org.infinispan.util.ByteArrayKey=org.infinispan.util.ByteArrayKey$Externalizer, class org.infinispan.container.entries.MortalCacheEntry=org.infinispan.container.entries.MortalCacheEntry$Externalizer, class org.infinispan.remoting.responses.UnsureResponse=org.infinispan.remoting.responses.UnsureResponse$Externalizer, class org.infinispan.atomic.ClearOperation=org.infinispan.atomic.ClearOperation$Externalizer, class org.infinispan.container.entries.TransientCacheValue=org.infinispan.container.entries.TransientCacheValue$Externalizer, class org.infinispan.atomic.AtomicHashMap=org.infinispan.atomic.AtomicHashMap$Externalizer, class org.infinispan.commands.tx.PrepareCommand=org.infinispan.marshall.exts.ReplicableCommandExternalizer, class org.infinispan.distribution.ch.TopologyAwareConsistentHash=org.infinispan.distribution.ch.TopologyAwareConsistentHash$Externalizer, class java.util.HashMap=org.infinispan.marshall.exts.MapExternalizer, class org.infinispan.commands.write.InvalidateL1Command=org.infinispan.marshall.exts.ReplicableCommandExternalizer, class org.infinispan.transaction.TransactionLog$LogEntry=org.infinispan.transaction.TransactionLog$LogEntry$Externalizer, class org.infinispan.remoting.responses.UnsuccessfulResponse=org.infinispan.remoting.responses.UnsuccessfulResponse$Externalizer, class org.infinispan.transaction.xa.recovery.SerializableXid=org.infinispan.transaction.xa.recovery.SerializableXid$XidExternalizer, class org.infinispan.commands.read.DistributedExecuteCommand=org.infinispan.marshall.exts.ReplicableCommandExternalizer, class org.infinispan.commands.write.EvictCommand=org.infinispan.marshall.exts.ReplicableCommandExternalizer, class org.infinispan.commands.remote.recovery.GetInDoubtTxInfoCommand=org.infinispan.marshall.exts.ReplicableCommandExternalizer, class org.infinispan.util.Immutables$ImmutableMapWrapper=org.infinispan.util.Immutables$ImmutableMapWrapperExternalizer, class org.infinispan.transaction.xa.recovery.RecoveryAwareDldGlobalTransaction=org.infinispan.transaction.xa.recovery.RecoveryAwareDldGlobalTransaction$Externalizer, class org.infinispan.commands.write.InvalidateCommand=org.infinispan.marshall.exts.ReplicableCommandExternalizer, class org.infinispan.container.entries.ImmortalCacheValue=org.infinispan.container.entries.ImmortalCacheValue$Externalizer, class org.infinispan.remoting.transport.jgroups.JGroupsTopologyAwareAddress=org.infinispan.remoting.transport.jgroups.JGroupsTopologyAwareAddress$Externalizer, class org.infinispan.remoting.responses.RequestIgnoredResponse=org.infinispan.remoting.responses.RequestIgnoredResponse$Externalizer, class org.infinispan.distribution.RemoteTransactionLogDetails=org.infinispan.distribution.RemoteTransactionLogDetails$Externalizer, class org.infinispan.distribution.ch.DefaultConsistentHash=org.infinispan.distribution.ch.DefaultConsistentHash$Externalizer, class org.infinispan.transaction.xa.recovery.RecoveryAwareGlobalTransaction=org.infinispan.transaction.xa.recovery.RecoveryAwareGlobalTransaction$Externalizer, class org.infinispan.remoting.responses.ExceptionResponse=org.infinispan.remoting.responses.ExceptionResponse$Externalizer, class org.infinispan.commands.write.RemoveCommand=org.infinispan.marshall.exts.ReplicableCommandExternalizer, class org.infinispan.container.entries.TransientMortalCacheValue=org.infinispan.container.entries.TransientMortalCacheValue$Externalizer, class org.infinispan.container.entries.TransientCacheEntry=org.infinispan.container.entries.TransientCacheEntry$Externalizer, class org.infinispan.transaction.xa.recovery.InDoubtTxInfoImpl=org.infinispan.transaction.xa.recovery.InDoubtTxInfoImpl$Externalizer, class org.infinispan.commands.tx.CommitCommand=org.infinispan.marshall.exts.ReplicableCommandExternalizer, class org.infinispan.remoting.responses.SuccessfulResponse=org.infinispan.remoting.responses.SuccessfulResponse$Externalizer, class org.infinispan.atomic.RemoveOperation=org.infinispan.atomic.RemoveOperation$Externalizer, class org.infinispan.commands.write.PutKeyValueCommand=org.infinispan.marshall.exts.ReplicableCommandExternalizer, class org.infinispan.remoting.responses.ExtendedResponse=org.infinispan.remoting.responses.ExtendedResponse$Externalizer, class org.infinispan.commands.remote.MultipleRpcCommand=org.infinispan.marshall.exts.ReplicableCommandExternalizer, class org.infinispan.remoting.transport.jgroups.JGroupsAddress=org.infinispan.remoting.transport.jgroups.JGroupsAddress$Externalizer, class org.infinispan.commands.remote.recovery.RemoveRecoveryInfoCommand=org.infinispan.marshall.exts.ReplicableCommandExternalizer, class org.infinispan.commands.RemoveCacheCommand=org.infinispan.marshall.exts.ReplicableCommandExternalizer, class org.infinispan.container.entries.MortalCacheValue=org.infinispan.container.entries.MortalCacheValue$Externalizer, class org.infinispan.commands.remote.recovery.CompleteTransactionCommand=org.infinispan.marshall.exts.ReplicableCommandExternalizer, class java.util.HashSet=org.infinispan.marshall.exts.SetExternalizer, class org.infinispan.commands.remote.ClusteredGetCommand=org.infinispan.marshall.exts.ReplicableCommandExternalizer, class org.infinispan.transaction.xa.DldGlobalTransaction=org.infinispan.transaction.xa.DldGlobalTransaction$Externalizer, class org.infinispan.distribution.ch.UnionConsistentHash=org.infinispan.distribution.ch.UnionConsistentHash$Externalizer, class java.util.TreeSet=org.infinispan.marshall.exts.SetExternalizer, class org.infinispan.container.entries.TransientMortalCacheEntry=org.infinispan.container.entries.TransientMortalCacheEntry$Externalizer} 2011-06-09 13:43:45,036 INFO [JGroupsTransport] (main) ISPN00078: Starting JGroups Channel 2011-06-09 13:43:45,036 INFO [JGroupsTransport] (main) ISPN00088: Unable to use any JGroups configuration mechanisms provided in properties {}. Using default JGroups configuration! 2011-06-09 13:43:45,126 INFO [JChannel] (main) JGroups version: 2.12.0.Final 2011-06-09 13:43:45,133 DEBUG [ClassConfigurator] (main) Using jg-magic-map.xml as magic number file and jg-protocol-ids.xml for protocol IDs 2011-06-09 13:43:45,451 DEBUG [GMS] (main) changed role to org.jgroups.protocols.pbcast.ClientGmsImpl 2011-06-09 13:43:45,601 DEBUG [Configurator] (main) set property UDP.diagnostics_addr to default value /224.0.75.75 2011-06-09 13:43:45,741 DEBUG [FRAG2] (main) received CONFIG event: {bind_addr=/127.0.0.1} 2011-06-09 13:43:45,858 TRACE [MessageDispatcher$ProtocolAdapter] (main) setting local_addr (null) to saudade-16366 2011-06-09 13:43:45,859 TRACE [STABLE] (main) stable task started 2011-06-09 13:43:45,859 DEBUG [UDP] (main) creating sockets 2011-06-09 13:43:45,859 DEBUG [UDP] (main) sockets will use interface 127.0.0.1 2011-06-09 13:43:45,863 WARN [UDP] (main) receive buffer of socket java.net.DatagramSocket@181b7c76 was set to 20MB, but the OS only allocated 65.51KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux) 2011-06-09 13:43:45,864 WARN [UDP] (main) receive buffer of socket java.net.MulticastSocket@34883357 was set to 25MB, but the OS only allocated 65.51KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux) 2011-06-09 13:43:45,864 DEBUG [UDP] (main) socket information: , mcast_addr=228.6.7.8:46655, bind_addr=/127.0.0.1, ttl=2 sock: bound to 127.0.0.1:52030, receive buffer size=65507, send buffer size=640000 mcast_sock: bound to 127.0.0.1:46655, send buffer size=640000, receive buffer size=65507 2011-06-09 13:43:45,868 DEBUG [UDP] (main) created unicast receiver thread 2011-06-09 13:43:45,868 DEBUG [UDP] (main) created multicast receiver thread 2011-06-09 13:43:45,873 TRACE [FD_SOCK] (FD_SOCK server socket acceptor,saudade-16366) waiting for client connections on /127.0.0.1:50599 2011-06-09 13:43:45,880 TRACE [UDP] (Timer-2,saudade-16366) sending msg to null, src=saudade-16366, headers are PING: [PING: type=GET_MBRS_REQ, cluster=Infinispan-Cluster, arg=own_addr=saudade-16366, view id=null, is_server=false, is_coord=false, logical_name=saudade-16366, physical_addrs=127.0.0.1:52030], UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:45,880 TRACE [UDP] (Timer-2,saudade-16366) looping back message [dst: , src: saudade-16366 (2 headers), size=0 bytes, flags=OOB] 2011-06-09 13:43:45,887 TRACE [UDP] (OOB-1,saudade-16366) received [dst: , src: saudade-16366 (2 headers), size=0 bytes, flags=OOB], headers are PING: [PING: type=GET_MBRS_REQ, cluster=Infinispan-Cluster, arg=own_addr=saudade-16366, view id=null, is_server=false, is_coord=false, logical_name=saudade-16366, physical_addrs=127.0.0.1:52030], UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:45,887 TRACE [PING] (OOB-1,saudade-16366) received GET_MBRS_REQ from saudade-16366, sending response [PING: type=GET_MBRS_RSP, arg=own_addr=saudade-16366, view id=null, is_server=false, is_coord=false, logical_name=saudade-16366, physical_addrs=127.0.0.1:52030] 2011-06-09 13:43:45,887 TRACE [UDP] (OOB-1,saudade-16366) sending msg to saudade-16366, src=saudade-16366, headers are PING: [PING: type=GET_MBRS_RSP, arg=own_addr=saudade-16366, view id=null, is_server=false, is_coord=false, logical_name=saudade-16366, physical_addrs=127.0.0.1:52030], UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:45,887 TRACE [UDP] (OOB-1,saudade-16366) looping back message [dst: saudade-16366, src: saudade-16366 (2 headers), size=0 bytes, flags=OOB] 2011-06-09 13:43:45,890 TRACE [UDP] (OOB-2,saudade-16366) received [dst: saudade-16366, src: saudade-16366 (2 headers), size=0 bytes, flags=OOB], headers are PING: [PING: type=GET_MBRS_RSP, arg=own_addr=saudade-16366, view id=null, is_server=false, is_coord=false, logical_name=saudade-16366, physical_addrs=127.0.0.1:52030], UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:45,890 TRACE [PING] (OOB-2,saudade-16366) received GET_MBRS_RSP from saudade-16366: own_addr=saudade-16366, view id=null, is_server=false, is_coord=false, logical_name=saudade-16366, physical_addrs=127.0.0.1:52030 2011-06-09 13:43:45,893 TRACE [UDP] (OOB-1,saudade-16366) received [dst: , src: saudade-16366 (2 headers), size=0 bytes, flags=OOB], headers are PING: [PING: type=GET_MBRS_REQ, cluster=Infinispan-Cluster, arg=own_addr=saudade-16366, view id=null, is_server=false, is_coord=false, logical_name=saudade-16366, physical_addrs=127.0.0.1:52030], UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:45,893 TRACE [UDP] (OOB-2,saudade-16366) received [dst: saudade-16366, src: a641248c-21bd-ab3e-d2ec-f612f4b460cc (2 headers), size=0 bytes, flags=OOB], headers are PING: [PING: type=GET_MBRS_RSP, arg=own_addr=a641248c-21bd-ab3e-d2ec-f612f4b460cc, view id=[a641248c-21bd-ab3e-d2ec-f612f4b460cc|0], is_server=true, is_coord=true, logical_name=saudade-59431, physical_addrs=127.0.0.1:55358], UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:45,893 TRACE [PING] (OOB-2,saudade-16366) received GET_MBRS_RSP from saudade-59431: own_addr=saudade-59431, view id=[saudade-59431|0], is_server=true, is_coord=true, logical_name=saudade-59431, physical_addrs=127.0.0.1:55358 2011-06-09 13:43:45,894 TRACE [PING] (main) discovery took 20 ms: responses: 2 total (1 servers (1 coord), 1 clients) 2011-06-09 13:43:45,894 DEBUG [GMS] (main) initial_mbrs are [own_addr=saudade-59431, view id=[saudade-59431|0], is_server=true, is_coord=true, logical_name=saudade-59431, physical_addrs=127.0.0.1:55358] 2011-06-09 13:43:45,894 DEBUG [GMS] (main) election results: {saudade-59431=1} 2011-06-09 13:43:45,894 DEBUG [GMS] (main) sending handleJoin(saudade-16366) to saudade-59431 2011-06-09 13:43:45,899 TRACE [UNICAST] (main) saudade-16366: created connection to saudade-59431 2011-06-09 13:43:45,900 TRACE [UNICAST] (main) saudade-16366 --> DATA(saudade-59431: #1, conn_id=0, first) 2011-06-09 13:43:45,901 TRACE [UDP] (main) sending msg to saudade-59431, src=saudade-16366, headers are GMS: GmsHeader[JOIN_REQ]: mbr=saudade-16366, UNICAST: DATA, seqno=1, first, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:45,917 TRACE [UDP] (Incoming-1,saudade-16366) received [dst: saudade-16366, src: saudade-59431 (2 headers), size=0 bytes], headers are UNICAST: ACK, seqno=1, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:45,917 TRACE [UNICAST] (Incoming-1,saudade-16366) saudade-16366 <-- ACK(saudade-59431: #1) 2011-06-09 13:43:45,983 TRACE [UDP] (OOB-2,saudade-16366) received [dst: , src: saudade-59431 (2 headers), size=0 bytes, flags=OOB], headers are FD_SOCK: I_HAVE_SOCK, mbr=saudade-59431, sock_addr=127.0.0.1:50596, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:45,985 TRACE [FD_SOCK] (OOB-2,saudade-16366) i-have-sock: saudade-59431 --> 127.0.0.1:50596 (cache is {saudade-59431=127.0.0.1:50596}) 2011-06-09 13:43:46,005 TRACE [UDP] (OOB-2,saudade-16366) received [dst: saudade-16366, src: saudade-59431 (2 headers), size=0 bytes, flags=OOB], headers are FD_SOCK: WHO_HAS_SOCK, mbr=saudade-16366, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:46,006 TRACE [FD_SOCK] (OOB-2,saudade-16366) who-has-sock saudade-16366 2011-06-09 13:43:46,006 TRACE [UDP] (OOB-2,saudade-16366) sending msg to saudade-59431, src=saudade-16366, headers are FD_SOCK: I_HAVE_SOCK, mbr=saudade-16366, sock_addr=127.0.0.1:50599, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:46,014 TRACE [UDP] (Incoming-2,saudade-16366) received [dst: saudade-16366, src: saudade-59431 (3 headers), size=0 bytes], headers are GMS: GmsHeader[JOIN_RSP]: join_rsp=view: [saudade-59431|1] [saudade-59431, saudade-16366], digest: saudade-16366: [0 : 0], saudade-59431: [0 : 1 (1)], UNICAST: DATA, seqno=1, conn_id=1, first, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:46,014 TRACE [UNICAST] (Incoming-2,saudade-16366) saudade-16366 <-- DATA(saudade-59431: #1, conn_id=1, first) 2011-06-09 13:43:46,017 TRACE [UNICAST] (Incoming-2,saudade-16366) saudade-16366: created receiver window for saudade-59431 at seqno=#1 for conn-id=1 2011-06-09 13:43:46,017 TRACE [UNICAST] (Incoming-2,saudade-16366) saudade-16366 --> ACK(saudade-59431: #1) 2011-06-09 13:43:46,017 TRACE [UDP] (Incoming-2,saudade-16366) sending msg to saudade-59431, src=saudade-16366, headers are UNICAST: ACK, seqno=1, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:46,043 TRACE [UDP] (Incoming-2,saudade-16366) received [dst: , src: saudade-59431 (3 headers), size=0 bytes], headers are GMS: GmsHeader[VIEW]: view=[saudade-59431|1] [saudade-59431, saudade-16366], NAKACK: [MSG, seqno=2], UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:46,043 TRACE [NAKACK] (Incoming-2,saudade-16366) message saudade-59431::2 was discarded (not yet server) 2011-06-09 13:43:46,053 TRACE [FD_SOCK] (FD_SOCK server socket acceptor,saudade-16366) accepted connection from /127.0.0.1:50600 2011-06-09 13:43:46,058 DEBUG [NAKACK] (main) [setDigest()] existing digest: [] new digest: saudade-16366: [0 : 0], saudade-59431: [0 : 2 (2)] resulting digest: saudade-16366: [0 : 0 (0)], saudade-59431: [0 : 2 (2)] 2011-06-09 13:43:46,059 DEBUG [GMS] (main) [saudade-16366]: JoinRsp=[saudade-59431|1] [saudade-59431, saudade-16366] [size=2] 2011-06-09 13:43:46,061 DEBUG [GMS] (main) new_view=[saudade-59431|1] [saudade-59431, saudade-16366] 2011-06-09 13:43:46,061 DEBUG [GMS] (main) saudade-16366: view is [saudade-59431|1] [saudade-59431, saudade-16366] 2011-06-09 13:43:46,062 TRACE [FD_SOCK] (FD_SOCK server socket acceptor,saudade-16366) waiting for client connections on /127.0.0.1:50599 2011-06-09 13:43:46,063 DEBUG [FD_SOCK] (main) VIEW_CHANGE received: [saudade-59431, saudade-16366] 2011-06-09 13:43:46,063 TRACE [UDP] (FD_SOCK pinger,saudade-16366) sending msg to null, src=saudade-16366, headers are FD_SOCK: I_HAVE_SOCK, mbr=saudade-16366, sock_addr=127.0.0.1:50599, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:46,063 TRACE [UDP] (FD_SOCK pinger,saudade-16366) looping back message [dst: , src: saudade-16366 (2 headers), size=0 bytes, flags=OOB] 2011-06-09 13:43:46,064 TRACE [UDP] (OOB-2,saudade-16366) received [dst: , src: saudade-16366 (2 headers), size=0 bytes, flags=OOB], headers are FD_SOCK: I_HAVE_SOCK, mbr=saudade-16366, sock_addr=127.0.0.1:50599, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:46,064 TRACE [FD_SOCK] (OOB-2,saudade-16366) i-have-sock: saudade-16366 --> 127.0.0.1:50599 (cache is {saudade-16366=127.0.0.1:50599, saudade-59431=127.0.0.1:50596}) 2011-06-09 13:43:46,064 TRACE [UDP] (OOB-1,saudade-16366) received [dst: , src: saudade-16366 (2 headers), size=0 bytes, flags=OOB], headers are FD_SOCK: I_HAVE_SOCK, mbr=saudade-16366, sock_addr=127.0.0.1:50599, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:46,065 TRACE [UDP] (FD_SOCK pinger,saudade-16366) sending msg to saudade-59431, src=saudade-16366, headers are FD_SOCK: GET_CACHE, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:46,066 TRACE [UDP] (OOB-1,saudade-16366) received [dst: saudade-16366, src: saudade-59431 (2 headers), size=0 bytes, flags=OOB], headers are FD_SOCK: GET_CACHE_RSP, cache={saudade-59431=127.0.0.1:50596, saudade-16366=127.0.0.1:50599}, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:46,067 TRACE [FD_SOCK] (FD_SOCK pinger,saudade-16366) got cache from saudade-59431: cache is {saudade-16366=127.0.0.1:50599, saudade-59431=127.0.0.1:50596} 2011-06-09 13:43:46,067 TRACE [FD_SOCK] (FD_SOCK pinger,saudade-16366) pinger_thread started 2011-06-09 13:43:46,067 DEBUG [FD_SOCK] (FD_SOCK pinger,saudade-16366) ping_dest is saudade-59431, pingable_mbrs=[saudade-59431, saudade-16366] 2011-06-09 13:43:46,068 TRACE [STABLE] (main) saudade-16366: resetting digest from NAKACK: [saudade-59431#2, saudade-16366#0] 2011-06-09 13:43:46,069 DEBUG [STABLE] (main) [ergonomics] setting max_bytes to 2MB (2 members) 2011-06-09 13:43:46,069 TRACE [UFC] (main) new membership: [saudade-59431, saudade-16366] 2011-06-09 13:43:46,070 TRACE [MFC] (main) new membership: [saudade-59431, saudade-16366] 2011-06-09 13:43:46,071 INFO [JGroupsTransport] (main) ISPN00094: Received new cluster view: [saudade-59431|1] [saudade-59431, saudade-16366] 2011-06-09 13:43:46,072 DEBUG [GMS] (main) saudade-16366: 2011-06-09 13:43:46,073 TRACE [UNICAST] (main) saudade-16366 --> DATA(saudade-59431: #2, conn_id=0) 2011-06-09 13:43:46,073 TRACE [FD_SOCK] (FD_SOCK pinger,saudade-16366) ping_dest=saudade-59431, ping_sock=Socket[addr=/127.0.0.1,port=50596,localport=50601], cache={saudade-16366=127.0.0.1:50599, saudade-59431=127.0.0.1:50596} 2011-06-09 13:43:46,073 TRACE [UDP] (main) sending msg to saudade-59431, src=saudade-16366, headers are GMS: GmsHeader[VIEW_ACK]: view=null, UNICAST: DATA, seqno=2, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:46,074 INFO [JGroupsTransport] (main) ISPN00079: Cache local address is saudade-16366, physical addresses are [127.0.0.1:52030] 2011-06-09 13:43:46,074 DEBUG [JGroupsTransport] (main) Waiting on view being accepted 2011-06-09 13:43:46,076 TRACE [UDP] (Incoming-2,saudade-16366) received [dst: saudade-16366, src: saudade-59431 (2 headers), size=0 bytes], headers are UNICAST: ACK, seqno=2, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:46,076 TRACE [UNICAST] (Incoming-2,saudade-16366) saudade-16366 <-- ACK(saudade-59431: #2) 2011-06-09 13:43:46,259 TRACE [UDP] (OOB-1,saudade-16366) received [dst: , src: saudade-59431 (3 headers), size=0 bytes, flags=OOB], headers are STABLE: [STABILITY]: digest is saudade-59431: [0 : 0 (0)], NAKACK: [MSG, seqno=3], UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:46,259 TRACE [NAKACK] (OOB-1,saudade-16366) saudade-16366: received saudade-59431#3 2011-06-09 13:43:46,260 TRACE [STABLE] (OOB-1,saudade-16366) saudade-16366: received stability msg from saudade-59431: [saudade-59431#0] 2011-06-09 13:43:46,260 DEBUG [STABLE] (OOB-1,saudade-16366) saudade-16366: received digest from saudade-59431 (digest=saudade-59431: [0 : 0 (0)]) which does not match my own digest (saudade-16366: [0 : 0 (0)], saudade-59431: [0 : 2 (2)]): ignoring digest and re-initializing own digest 2011-06-09 13:43:46,260 TRACE [STABLE] (OOB-1,saudade-16366) saudade-16366: resetting digest from NAKACK: [saudade-59431#2, saudade-16366#0] 2011-06-09 13:43:46,315 TRACE [TransactionFactory] (main) Setting factory enum to NODLD_NORECOVERY_XA 2011-06-09 13:43:46,315 DEBUG [InterceptorChain] (main) Interceptor chain size: 6 2011-06-09 13:43:46,315 DEBUG [InterceptorChain] (main) Interceptor chain is: >> org.infinispan.interceptors.InvocationContextInterceptor >> org.infinispan.interceptors.TxInterceptor >> org.infinispan.interceptors.NotificationInterceptor >> org.infinispan.interceptors.LockingInterceptor >> org.infinispan.interceptors.ReplicationInterceptor >> org.infinispan.interceptors.CallInterceptor 2011-06-09 13:43:46,317 TRACE [StateTransferManagerImpl] (main) Data container is 63f8247d 2011-06-09 13:43:46,317 DEBUG [StateTransferManagerImpl] (main) Initiating state transfer process 2011-06-09 13:43:46,318 INFO [RpcManagerImpl] (main) ISPN00074: Trying to fetch state from saudade-59431 2011-06-09 13:43:46,319 DEBUG [STREAMING_STATE_TRANSFER] (main) GET_STATE: asking saudade-59431 for state, passing down a SUSPEND_STABLE event, timeout=10000 2011-06-09 13:43:46,320 DEBUG [STABLE] (main) suspending message garbage collection 2011-06-09 13:43:46,320 DEBUG [STABLE] (main) resume task started, max_suspend_time=11000 2011-06-09 13:43:46,320 TRACE [UNICAST] (main) saudade-16366 --> DATA(saudade-59431: #3, conn_id=0) 2011-06-09 13:43:46,320 TRACE [UDP] (main) sending msg to saudade-59431, src=saudade-16366, headers are STREAMING_STATE_TRANSFER: type=STATE_REQ, sender=saudade-16366 id=0, UNICAST: DATA, seqno=3, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:46,326 TRACE [UDP] (Incoming-2,saudade-16366) received [dst: saudade-16366, src: saudade-59431 (2 headers), size=0 bytes], headers are UNICAST: ACK, seqno=3, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:46,326 TRACE [UNICAST] (Incoming-2,saudade-16366) saudade-16366 <-- ACK(saudade-59431: #3) 2011-06-09 13:43:46,335 TRACE [UDP] (Incoming-1,saudade-16366) received [dst: saudade-16366, src: saudade-59431 (3 headers), size=0 bytes], headers are STREAMING_STATE_TRANSFER: type=STATE_RSP, sender=saudade-59431 id=0, digest=saudade-16366: [0 : 0 (0)], saudade-59431: [0 : 3 (3)], UNICAST: DATA, seqno=2, conn_id=1, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:46,335 TRACE [UNICAST] (Incoming-1,saudade-16366) saudade-16366 <-- DATA(saudade-59431: #2, conn_id=1) 2011-06-09 13:43:46,335 TRACE [UNICAST] (Incoming-1,saudade-16366) saudade-16366 --> ACK(saudade-59431: #2) 2011-06-09 13:43:46,336 TRACE [UDP] (Incoming-1,saudade-16366) sending msg to saudade-59431, src=saudade-16366, headers are UNICAST: ACK, seqno=2, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:46,336 TRACE [BARRIER] (Incoming-1,saudade-16366) barrier was closed 2011-06-09 13:43:46,338 DEBUG [NAKACK] (Incoming-1,saudade-16366) [overwriteDigest()] existing digest: saudade-16366: [0 : 0 (0)], saudade-59431: [0 : 3 (3)] new digest: saudade-16366: [0 : 0 (0)], saudade-59431: [0 : 3 (3)] resulting digest: saudade-16366: [0 : 0 (0)], saudade-59431: [0 : 3 (3)] 2011-06-09 13:43:46,338 DEBUG [STREAMING_STATE_TRANSFER] (Incoming-1,saudade-16366) Connecting to state provider /127.0.0.1:50602, original buffer size was 262140 and was reset to 8192 2011-06-09 13:43:46,339 DEBUG [STREAMING_STATE_TRANSFER] (Incoming-1,saudade-16366) Connected to state provider, my end of the socket is /127.0.0.1:50603 passing inputstream up... 2011-06-09 13:43:46,345 TRACE [JGroupsTransport] (Incoming-1,saudade-16366) Received state for cache named 'test'. Attempting to apply state. 2011-06-09 13:43:46,345 DEBUG [StateTransferManagerImpl] (Incoming-1,saudade-16366) Applying state 2011-06-09 13:43:46,380 TRACE [JBossMarshaller] (Incoming-1,saudade-16366) Start unmarshaller after retrieving marshaller from thread local 2011-06-09 13:43:46,408 TRACE [VersionAwareMarshaller] (Incoming-1,saudade-16366) Read version 500 2011-06-09 13:43:46,417 TRACE [StateTransferManagerImpl] (Incoming-1,saudade-16366) Integrating transaction log 2011-06-09 13:43:46,417 TRACE [StateTransferManagerImpl] (Incoming-1,saudade-16366) Applying commit log 2011-06-09 13:43:46,418 TRACE [StateTransferManagerImpl] (Incoming-1,saudade-16366) Finished applying commit log 2011-06-09 13:43:46,418 TRACE [JGroupsTransport] (Incoming-1,saudade-16366) dests=[saudade-59431], command=StateTransferControlCommand{enabled=true}, mode=SYNCHRONOUS, timeout=10000 2011-06-09 13:43:46,420 TRACE [CommandAwareRpcDispatcher] (Incoming-1,saudade-16366) Replication task sending StateTransferControlCommand{enabled=true} to addresses [saudade-59431] 2011-06-09 13:43:46,481 TRACE [JBossMarshaller] (Incoming-1,saudade-16366) Start marshaller after retrieving marshaller from thread local 2011-06-09 13:43:46,481 TRACE [VersionAwareMarshaller] (Incoming-1,saudade-16366) Wrote version 500 2011-06-09 13:43:46,511 TRACE [JBossMarshaller] (Incoming-1,saudade-16366) Stop marshaller 2011-06-09 13:43:46,555 TRACE [Request] (Incoming-1,saudade-16366) sending request (id=2) 2011-06-09 13:43:46,555 TRACE [UNICAST] (Incoming-1,saudade-16366) saudade-16366 --> DATA(saudade-59431: #4, conn_id=0) 2011-06-09 13:43:46,555 TRACE [UDP] (Incoming-1,saudade-16366) sending msg to saudade-59431, src=saudade-16366, headers are RequestCorrelator: id=201, type=REQ, id=2, rsp_expected=true, UNICAST: DATA, seqno=4, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:46,669 TRACE [UDP] (OOB-1,saudade-16366) received [dst: saudade-16366, src: saudade-59431 (3 headers), size=4 bytes, flags=OOB|DONT_BUNDLE|NO_FC], headers are RequestCorrelator: id=201, type=RSP, id=2, rsp_expected=false, UNICAST: DATA, seqno=3, conn_id=1, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:46,670 TRACE [UDP] (Incoming-2,saudade-16366) received [dst: saudade-16366, src: saudade-59431 (2 headers), size=0 bytes], headers are UNICAST: ACK, seqno=4, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:46,856 TRACE [UNICAST] (Timer-2,saudade-16366) saudade-16366 --> XMIT(saudade-59431: #4) 2011-06-09 13:43:46,856 TRACE [UDP] (Timer-2,saudade-16366) sending msg to saudade-59431, src=saudade-16366, headers are RequestCorrelator: id=201, type=REQ, id=2, rsp_expected=true, UNICAST: DATA, seqno=4, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:46,970 TRACE [UDP] (OOB-2,saudade-16366) received [dst: saudade-16366, src: saudade-59431 (3 headers), size=4 bytes, flags=OOB|DONT_BUNDLE|NO_FC], headers are RequestCorrelator: id=201, type=RSP, id=2, rsp_expected=false, UNICAST: DATA, seqno=3, conn_id=1, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:47,456 TRACE [UNICAST] (Timer-2,saudade-16366) saudade-16366 --> XMIT(saudade-59431: #4) 2011-06-09 13:43:47,457 TRACE [UDP] (Timer-2,saudade-16366) sending msg to saudade-59431, src=saudade-16366, headers are RequestCorrelator: id=201, type=REQ, id=2, rsp_expected=true, UNICAST: DATA, seqno=4, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:47,571 TRACE [UDP] (OOB-3,saudade-16366) received [dst: saudade-16366, src: saudade-59431 (3 headers), size=4 bytes, flags=OOB|DONT_BUNDLE|NO_FC], headers are RequestCorrelator: id=201, type=RSP, id=2, rsp_expected=false, UNICAST: DATA, seqno=3, conn_id=1, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:48,657 TRACE [UNICAST] (Timer-3,saudade-16366) saudade-16366 --> XMIT(saudade-59431: #4) 2011-06-09 13:43:48,658 TRACE [UDP] (Timer-3,saudade-16366) sending msg to saudade-59431, src=saudade-16366, headers are RequestCorrelator: id=201, type=REQ, id=2, rsp_expected=true, UNICAST: DATA, seqno=4, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:48,773 TRACE [UDP] (OOB-4,saudade-16366) received [dst: saudade-16366, src: saudade-59431 (3 headers), size=4 bytes, flags=OOB|DONT_BUNDLE|NO_FC], headers are RequestCorrelator: id=201, type=RSP, id=2, rsp_expected=false, UNICAST: DATA, seqno=3, conn_id=1, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:48,976 TRACE [UDP] (OOB-5,saudade-16366) received [dst: , src: saudade-59431 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:49,068 TRACE [UDP] (Timer-4,saudade-16366) sending msg to null, src=saudade-16366, headers are FD_ALL: heartbeat, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:49,068 TRACE [UDP] (Timer-4,saudade-16366) looping back message [dst: , src: saudade-16366 (2 headers), size=0 bytes, flags=OOB] 2011-06-09 13:43:49,069 TRACE [UDP] (OOB-5,saudade-16366) received [dst: , src: saudade-16366 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:49,098 TRACE [UDP] (OOB-5,saudade-16366) received [dst: , src: saudade-16366 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:49,858 TRACE [UNICAST] (Timer-2,saudade-16366) saudade-16366 --> XMIT(saudade-59431: #4) 2011-06-09 13:43:49,858 TRACE [UDP] (Timer-2,saudade-16366) sending msg to saudade-59431, src=saudade-16366, headers are RequestCorrelator: id=201, type=REQ, id=2, rsp_expected=true, UNICAST: DATA, seqno=4, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:49,989 TRACE [UDP] (OOB-5,saudade-16366) received [dst: saudade-16366, src: saudade-59431 (3 headers), size=4 bytes, flags=OOB|DONT_BUNDLE|NO_FC], headers are RequestCorrelator: id=201, type=RSP, id=2, rsp_expected=false, UNICAST: DATA, seqno=3, conn_id=1, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:51,060 TRACE [UNICAST] (Timer-3,saudade-16366) saudade-16366 --> XMIT(saudade-59431: #4) 2011-06-09 13:43:51,060 TRACE [UDP] (Timer-3,saudade-16366) sending msg to saudade-59431, src=saudade-16366, headers are RequestCorrelator: id=201, type=REQ, id=2, rsp_expected=true, UNICAST: DATA, seqno=4, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:51,168 TRACE [UDP] (OOB-6,saudade-16366) received [dst: , src: saudade-59431 (3 headers), size=0 bytes, flags=OOB], headers are STABLE: [STABLE_GOSSIP]: digest is saudade-16366: [0 : 0 (0)], saudade-59431: [0 : 3 (3)], NAKACK: [MSG, seqno=4], UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:51,192 TRACE [UDP] (OOB-7,saudade-16366) received [dst: saudade-16366, src: saudade-59431 (3 headers), size=4 bytes, flags=OOB|DONT_BUNDLE|NO_FC], headers are RequestCorrelator: id=201, type=RSP, id=2, rsp_expected=false, UNICAST: DATA, seqno=3, conn_id=1, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:51,976 TRACE [UDP] (OOB-8,saudade-16366) received [dst: , src: saudade-59431 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:52,069 TRACE [UDP] (Timer-4,saudade-16366) sending msg to null, src=saudade-16366, headers are FD_ALL: heartbeat, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:52,069 TRACE [UDP] (Timer-4,saudade-16366) looping back message [dst: , src: saudade-16366 (2 headers), size=0 bytes, flags=OOB] 2011-06-09 13:43:52,070 TRACE [UDP] (OOB-8,saudade-16366) received [dst: , src: saudade-16366 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:52,070 TRACE [UDP] (OOB-8,saudade-16366) received [dst: , src: saudade-16366 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:52,260 TRACE [UNICAST] (Timer-2,saudade-16366) saudade-16366 --> XMIT(saudade-59431: #4) 2011-06-09 13:43:52,261 TRACE [UDP] (Timer-2,saudade-16366) sending msg to saudade-59431, src=saudade-16366, headers are RequestCorrelator: id=201, type=REQ, id=2, rsp_expected=true, UNICAST: DATA, seqno=4, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:52,393 TRACE [UDP] (OOB-8,saudade-16366) received [dst: saudade-16366, src: saudade-59431 (3 headers), size=4 bytes, flags=OOB|DONT_BUNDLE|NO_FC], headers are RequestCorrelator: id=201, type=RSP, id=2, rsp_expected=false, UNICAST: DATA, seqno=3, conn_id=1, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:53,492 TRACE [UNICAST] (Timer-3,saudade-16366) saudade-16366 --> XMIT(saudade-59431: #4) 2011-06-09 13:43:53,492 TRACE [UDP] (Timer-3,saudade-16366) sending msg to saudade-59431, src=saudade-16366, headers are RequestCorrelator: id=201, type=REQ, id=2, rsp_expected=true, UNICAST: DATA, seqno=4, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:53,594 TRACE [UDP] (OOB-9,saudade-16366) received [dst: saudade-16366, src: saudade-59431 (3 headers), size=4 bytes, flags=OOB|DONT_BUNDLE|NO_FC], headers are RequestCorrelator: id=201, type=RSP, id=2, rsp_expected=false, UNICAST: DATA, seqno=3, conn_id=1, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:54,692 TRACE [UNICAST] (Timer-4,saudade-16366) saudade-16366 --> XMIT(saudade-59431: #4) 2011-06-09 13:43:54,693 TRACE [UDP] (Timer-4,saudade-16366) sending msg to saudade-59431, src=saudade-16366, headers are RequestCorrelator: id=201, type=REQ, id=2, rsp_expected=true, UNICAST: DATA, seqno=4, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:54,830 TRACE [UDP] (OOB-10,saudade-16366) received [dst: saudade-16366, src: saudade-59431 (3 headers), size=4 bytes, flags=OOB|DONT_BUNDLE|NO_FC], headers are RequestCorrelator: id=201, type=RSP, id=2, rsp_expected=false, UNICAST: DATA, seqno=3, conn_id=1, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:54,990 TRACE [UDP] (OOB-11,saudade-16366) received [dst: , src: saudade-59431 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:55,070 TRACE [UDP] (Timer-5,saudade-16366) sending msg to null, src=saudade-16366, headers are FD_ALL: heartbeat, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:55,070 TRACE [UDP] (Timer-5,saudade-16366) looping back message [dst: , src: saudade-16366 (2 headers), size=0 bytes, flags=OOB] 2011-06-09 13:43:55,071 TRACE [UDP] (OOB-11,saudade-16366) received [dst: , src: saudade-16366 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:55,071 TRACE [UDP] (OOB-11,saudade-16366) received [dst: , src: saudade-16366 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:55,860 TRACE [STABLE] (Timer-3,saudade-16366) stable task will not run as suspended=true 2011-06-09 13:43:55,893 TRACE [UNICAST] (Timer-4,saudade-16366) saudade-16366 --> XMIT(saudade-59431: #4) 2011-06-09 13:43:55,893 TRACE [UDP] (Timer-4,saudade-16366) sending msg to saudade-59431, src=saudade-16366, headers are RequestCorrelator: id=201, type=REQ, id=2, rsp_expected=true, UNICAST: DATA, seqno=4, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:56,009 TRACE [UDP] (OOB-11,saudade-16366) received [dst: saudade-16366, src: saudade-59431 (3 headers), size=4 bytes, flags=OOB|DONT_BUNDLE|NO_FC], headers are RequestCorrelator: id=201, type=RSP, id=2, rsp_expected=false, UNICAST: DATA, seqno=3, conn_id=1, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:56,435 TRACE [STABLE] (main) saudade-16366: resetting digest from NAKACK: [saudade-59431#3, saudade-16366#0] 2011-06-09 13:43:56,435 DEBUG [STABLE] (main) resuming message garbage collection 2011-06-09 13:43:56,611 TRACE [RpcManagerImpl] (Incoming-1,saudade-16366) replication exception: org.infinispan.util.concurrent.TimeoutException: Timed out after 10 seconds waiting for a response from saudade-59431 at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$ReplicationTask.call(CommandAwareRpcDispatcher.java:267) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:116) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:436) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:131) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:155) at org.infinispan.statetransfer.StateTransferManagerImpl.mimicPartialFlushViaRPC(StateTransferManagerImpl.java:301) at org.infinispan.statetransfer.StateTransferManagerImpl.applyTransactionLog(StateTransferManagerImpl.java:254) at org.infinispan.statetransfer.StateTransferManagerImpl.applyState(StateTransferManagerImpl.java:322) at org.infinispan.remoting.InboundInvocationHandlerImpl.applyState(InboundInvocationHandlerImpl.java:230) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.setState(JGroupsTransport.java:602) at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUpEvent(MessageDispatcher.java:711) at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:771) at org.jgroups.JChannel.up(JChannel.java:1441) at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1074) at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.connectToStateProvider(STREAMING_STATE_TRANSFER.java:523) at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.handleStateRsp(STREAMING_STATE_TRANSFER.java:462) at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.up(STREAMING_STATE_TRANSFER.java:223) at org.jgroups.protocols.FRAG2.up(FRAG2.java:189) at org.jgroups.protocols.FlowControl.up(FlowControl.java:418) at org.jgroups.protocols.FlowControl.up(FlowControl.java:400) at org.jgroups.protocols.pbcast.GMS.up(GMS.java:891) at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:246) at org.jgroups.protocols.UNICAST.handleDataReceived(UNICAST.java:613) at org.jgroups.protocols.UNICAST.up(UNICAST.java:294) at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:703) at org.jgroups.protocols.BARRIER.up(BARRIER.java:119) at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:177) at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:275) at org.jgroups.protocols.MERGE2.up(MERGE2.java:209) at org.jgroups.protocols.Discovery.up(Discovery.java:291) at org.jgroups.protocols.PING.up(PING.java:66) at org.jgroups.protocols.TP.passMessageUp(TP.java:1102) at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1658) at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1640) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:637) 2011-06-09 13:43:56,615 TRACE [JBossMarshaller] (Incoming-1,saudade-16366) Stop unmarshaller 2011-06-09 13:43:56,615 ERROR [JGroupsTransport] (Incoming-1,saudade-16366) ISPN00096: Caught while requesting or applying state org.infinispan.statetransfer.StateTransferException: org.infinispan.util.concurrent.TimeoutException: Timed out after 10 seconds waiting for a response from saudade-59431 at org.infinispan.statetransfer.StateTransferManagerImpl.applyState(StateTransferManagerImpl.java:333) at org.infinispan.remoting.InboundInvocationHandlerImpl.applyState(InboundInvocationHandlerImpl.java:230) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.setState(JGroupsTransport.java:602) at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUpEvent(MessageDispatcher.java:711) at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:771) at org.jgroups.JChannel.up(JChannel.java:1441) at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1074) at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.connectToStateProvider(STREAMING_STATE_TRANSFER.java:523) at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.handleStateRsp(STREAMING_STATE_TRANSFER.java:462) at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.up(STREAMING_STATE_TRANSFER.java:223) at org.jgroups.protocols.FRAG2.up(FRAG2.java:189) at org.jgroups.protocols.FlowControl.up(FlowControl.java:418) at org.jgroups.protocols.FlowControl.up(FlowControl.java:400) at org.jgroups.protocols.pbcast.GMS.up(GMS.java:891) at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:246) at org.jgroups.protocols.UNICAST.handleDataReceived(UNICAST.java:613) at org.jgroups.protocols.UNICAST.up(UNICAST.java:294) at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:703) at org.jgroups.protocols.BARRIER.up(BARRIER.java:119) at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:177) at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:275) at org.jgroups.protocols.MERGE2.up(MERGE2.java:209) at org.jgroups.protocols.Discovery.up(Discovery.java:291) at org.jgroups.protocols.PING.up(PING.java:66) at org.jgroups.protocols.TP.passMessageUp(TP.java:1102) at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1658) at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1640) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:637) Caused by: org.infinispan.util.concurrent.TimeoutException: Timed out after 10 seconds waiting for a response from saudade-59431 at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$ReplicationTask.call(CommandAwareRpcDispatcher.java:267) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:116) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:436) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:131) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:155) at org.infinispan.statetransfer.StateTransferManagerImpl.mimicPartialFlushViaRPC(StateTransferManagerImpl.java:301) at org.infinispan.statetransfer.StateTransferManagerImpl.applyTransactionLog(StateTransferManagerImpl.java:254) at org.infinispan.statetransfer.StateTransferManagerImpl.applyState(StateTransferManagerImpl.java:322) ... 29 more 2011-06-09 13:43:56,616 DEBUG [STREAMING_STATE_TRANSFER] (Incoming-1,saudade-16366) State reader is closing the socket 2011-06-09 13:43:56,617 DEBUG [STREAMING_STATE_TRANSFER] (Incoming-1,saudade-16366) STATE_TRANSFER_INPUTSTREAM_CLOSED received 2011-06-09 13:43:56,617 TRACE [STABLE] (Incoming-1,saudade-16366) saudade-16366: resetting digest from NAKACK: [saudade-59431#3, saudade-16366#0] 2011-06-09 13:43:56,617 DEBUG [STABLE] (Incoming-1,saudade-16366) resuming message garbage collection 2011-06-09 13:43:56,618 TRACE [BARRIER] (Incoming-1,saudade-16366) barrier was opened 2011-06-09 13:43:56,619 TRACE [UNICAST] (OOB-1,saudade-16366) saudade-16366 <-- DATA(saudade-59431: #3, conn_id=1) 2011-06-09 13:43:56,619 TRACE [JBossMarshaller] (OOB-1,saudade-16366) Start unmarshaller after retrieving marshaller from thread local 2011-06-09 13:43:56,619 TRACE [VersionAwareMarshaller] (OOB-1,saudade-16366) Read version 500 2011-06-09 13:43:56,619 TRACE [JBossMarshaller] (OOB-1,saudade-16366) Stop unmarshaller 2011-06-09 13:43:56,619 TRACE [Request] (OOB-1,saudade-16366) received response for request 2, sender=saudade-59431, val=null 2011-06-09 13:43:56,619 TRACE [UNICAST] (OOB-1,saudade-16366) saudade-16366 --> ACK(saudade-59431: #3) 2011-06-09 13:43:56,620 TRACE [UDP] (OOB-1,saudade-16366) sending msg to saudade-59431, src=saudade-16366, headers are UNICAST: ACK, seqno=3, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:56,620 TRACE [UNICAST] (Incoming-2,saudade-16366) saudade-16366 <-- ACK(saudade-59431: #4) 2011-06-09 13:43:56,620 TRACE [UNICAST] (OOB-2,saudade-16366) saudade-16366 <-- DATA(saudade-59431: #3, conn_id=1) 2011-06-09 13:43:56,623 WARN [RpcManagerImpl] (main) ISPN00075: Could not find available peer for state, backing off and retrying 2011-06-09 13:43:56,624 TRACE [UNICAST] (OOB-3,saudade-16366) saudade-16366 <-- DATA(saudade-59431: #3, conn_id=1) 2011-06-09 13:43:56,625 TRACE [UNICAST] (OOB-4,saudade-16366) saudade-16366 <-- DATA(saudade-59431: #3, conn_id=1) 2011-06-09 13:43:56,625 TRACE [UNICAST] (OOB-5,saudade-16366) saudade-16366 <-- DATA(saudade-59431: #3, conn_id=1) 2011-06-09 13:43:56,625 TRACE [NAKACK] (OOB-6,saudade-16366) saudade-16366: received saudade-59431#4 2011-06-09 13:43:56,642 TRACE [STABLE] (OOB-6,saudade-16366) saudade-16366: handling digest from saudade-59431 (0 votes): mine: [saudade-59431#3, saudade-16366#0] other: [saudade-59431#3, saudade-16366#0] result: [saudade-59431#3, saudade-16366#0] 2011-06-09 13:43:56,642 TRACE [UNICAST] (OOB-11,saudade-16366) saudade-16366 <-- DATA(saudade-59431: #3, conn_id=1) 2011-06-09 13:43:56,642 TRACE [UNICAST] (OOB-10,saudade-16366) saudade-16366 <-- DATA(saudade-59431: #3, conn_id=1) 2011-06-09 13:43:56,641 TRACE [UNICAST] (OOB-9,saudade-16366) saudade-16366 <-- DATA(saudade-59431: #3, conn_id=1) 2011-06-09 13:43:56,641 TRACE [UNICAST] (OOB-8,saudade-16366) saudade-16366 <-- DATA(saudade-59431: #3, conn_id=1) 2011-06-09 13:43:56,640 TRACE [UNICAST] (OOB-7,saudade-16366) saudade-16366 <-- DATA(saudade-59431: #3, conn_id=1) 2011-06-09 13:43:57,623 INFO [RpcManagerImpl] (main) ISPN00074: Trying to fetch state from saudade-59431 2011-06-09 13:43:57,624 DEBUG [STREAMING_STATE_TRANSFER] (main) GET_STATE: asking saudade-59431 for state, passing down a SUSPEND_STABLE event, timeout=10000 2011-06-09 13:43:57,624 DEBUG [STABLE] (main) suspending message garbage collection 2011-06-09 13:43:57,624 DEBUG [STABLE] (main) resume task started, max_suspend_time=11000 2011-06-09 13:43:57,624 TRACE [UNICAST] (main) saudade-16366 --> DATA(saudade-59431: #5, conn_id=0) 2011-06-09 13:43:57,625 TRACE [UDP] (main) sending msg to saudade-59431, src=saudade-16366, headers are STREAMING_STATE_TRANSFER: type=STATE_REQ, sender=saudade-16366 id=0, UNICAST: DATA, seqno=5, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:57,627 TRACE [UDP] (Incoming-2,saudade-16366) received [dst: saudade-16366, src: saudade-59431 (2 headers), size=0 bytes], headers are UNICAST: ACK, seqno=5, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:57,627 TRACE [UNICAST] (Incoming-2,saudade-16366) saudade-16366 <-- ACK(saudade-59431: #5) 2011-06-09 13:43:57,628 TRACE [UDP] (Incoming-2,saudade-16366) received [dst: saudade-16366, src: saudade-59431 (3 headers), size=0 bytes], headers are STREAMING_STATE_TRANSFER: type=STATE_RSP, sender=saudade-59431 id=0, digest=saudade-16366: [0 : 0 (0)], saudade-59431: [0 : 4 (4)], UNICAST: DATA, seqno=4, conn_id=1, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:57,628 TRACE [UNICAST] (Incoming-2,saudade-16366) saudade-16366 <-- DATA(saudade-59431: #4, conn_id=1) 2011-06-09 13:43:57,628 TRACE [UNICAST] (Incoming-2,saudade-16366) saudade-16366 --> ACK(saudade-59431: #4) 2011-06-09 13:43:57,629 TRACE [UDP] (Incoming-2,saudade-16366) sending msg to saudade-59431, src=saudade-16366, headers are UNICAST: ACK, seqno=4, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:57,629 TRACE [BARRIER] (Incoming-2,saudade-16366) barrier was closed 2011-06-09 13:43:57,629 DEBUG [NAKACK] (Incoming-2,saudade-16366) [overwriteDigest()] existing digest: saudade-16366: [0 : 0 (0)], saudade-59431: [0 : 4 (4)] new digest: saudade-16366: [0 : 0 (0)], saudade-59431: [0 : 4 (4)] resulting digest: saudade-16366: [0 : 0 (0)], saudade-59431: [0 : 4 (4)] 2011-06-09 13:43:57,629 DEBUG [STREAMING_STATE_TRANSFER] (Incoming-2,saudade-16366) Connecting to state provider /127.0.0.1:50602, original buffer size was 262140 and was reset to 8192 2011-06-09 13:43:57,630 DEBUG [STREAMING_STATE_TRANSFER] (Incoming-2,saudade-16366) Connected to state provider, my end of the socket is /127.0.0.1:50607 passing inputstream up... 2011-06-09 13:43:57,630 TRACE [JGroupsTransport] (Incoming-2,saudade-16366) Received state for cache named 'test'. Attempting to apply state. 2011-06-09 13:43:57,630 DEBUG [StateTransferManagerImpl] (Incoming-2,saudade-16366) Applying state 2011-06-09 13:43:57,631 TRACE [JBossMarshaller] (Incoming-2,saudade-16366) Start unmarshaller after retrieving marshaller from thread local 2011-06-09 13:43:57,633 TRACE [VersionAwareMarshaller] (Incoming-2,saudade-16366) Read version 500 2011-06-09 13:43:57,633 TRACE [StateTransferManagerImpl] (Incoming-2,saudade-16366) Integrating transaction log 2011-06-09 13:43:57,633 TRACE [StateTransferManagerImpl] (Incoming-2,saudade-16366) Applying commit log 2011-06-09 13:43:57,633 TRACE [StateTransferManagerImpl] (Incoming-2,saudade-16366) Finished applying commit log 2011-06-09 13:43:57,633 TRACE [JGroupsTransport] (Incoming-2,saudade-16366) dests=[saudade-59431], command=StateTransferControlCommand{enabled=true}, mode=SYNCHRONOUS, timeout=10000 2011-06-09 13:43:57,633 TRACE [CommandAwareRpcDispatcher] (Incoming-2,saudade-16366) Replication task sending StateTransferControlCommand{enabled=true} to addresses [saudade-59431] 2011-06-09 13:43:57,634 TRACE [JBossMarshaller] (Incoming-2,saudade-16366) Start marshaller after retrieving marshaller from thread local 2011-06-09 13:43:57,634 TRACE [VersionAwareMarshaller] (Incoming-2,saudade-16366) Wrote version 500 2011-06-09 13:43:57,634 TRACE [JBossMarshaller] (Incoming-2,saudade-16366) Stop marshaller 2011-06-09 13:43:57,634 TRACE [Request] (Incoming-2,saudade-16366) sending request (id=3) 2011-06-09 13:43:57,634 TRACE [UNICAST] (Incoming-2,saudade-16366) saudade-16366 --> DATA(saudade-59431: #6, conn_id=0) 2011-06-09 13:43:57,634 TRACE [UDP] (Incoming-2,saudade-16366) sending msg to saudade-59431, src=saudade-16366, headers are RequestCorrelator: id=201, type=REQ, id=3, rsp_expected=true, UNICAST: DATA, seqno=6, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:57,637 TRACE [UDP] (OOB-7,saudade-16366) received [dst: saudade-16366, src: saudade-59431 (3 headers), size=4 bytes, flags=OOB|DONT_BUNDLE|NO_FC], headers are RequestCorrelator: id=201, type=RSP, id=3, rsp_expected=false, UNICAST: DATA, seqno=5, conn_id=1, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:57,637 TRACE [UDP] (Incoming-1,saudade-16366) received [dst: saudade-16366, src: saudade-59431 (2 headers), size=0 bytes], headers are UNICAST: ACK, seqno=6, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:57,934 TRACE [UNICAST] (Timer-4,saudade-16366) saudade-16366 --> XMIT(saudade-59431: #6) 2011-06-09 13:43:57,935 TRACE [UDP] (Timer-4,saudade-16366) sending msg to saudade-59431, src=saudade-16366, headers are RequestCorrelator: id=201, type=REQ, id=3, rsp_expected=true, UNICAST: DATA, seqno=6, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:57,937 TRACE [UDP] (OOB-8,saudade-16366) received [dst: saudade-16366, src: saudade-59431 (3 headers), size=4 bytes, flags=OOB|DONT_BUNDLE|NO_FC], headers are RequestCorrelator: id=201, type=RSP, id=3, rsp_expected=false, UNICAST: DATA, seqno=5, conn_id=1, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:57,985 TRACE [UDP] (OOB-9,saudade-16366) received [dst: , src: saudade-59431 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:58,071 TRACE [UDP] (Timer-5,saudade-16366) sending msg to null, src=saudade-16366, headers are FD_ALL: heartbeat, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:58,072 TRACE [UDP] (Timer-5,saudade-16366) looping back message [dst: , src: saudade-16366 (2 headers), size=0 bytes, flags=OOB] 2011-06-09 13:43:58,072 TRACE [UDP] (OOB-9,saudade-16366) received [dst: , src: saudade-16366 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:58,073 TRACE [UDP] (OOB-9,saudade-16366) received [dst: , src: saudade-16366 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:58,535 TRACE [UNICAST] (Timer-3,saudade-16366) saudade-16366 --> XMIT(saudade-59431: #6) 2011-06-09 13:43:58,536 TRACE [UDP] (Timer-3,saudade-16366) sending msg to saudade-59431, src=saudade-16366, headers are RequestCorrelator: id=201, type=REQ, id=3, rsp_expected=true, UNICAST: DATA, seqno=6, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:58,538 TRACE [UDP] (OOB-9,saudade-16366) received [dst: saudade-16366, src: saudade-59431 (3 headers), size=4 bytes, flags=OOB|DONT_BUNDLE|NO_FC], headers are RequestCorrelator: id=201, type=RSP, id=3, rsp_expected=false, UNICAST: DATA, seqno=5, conn_id=1, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:59,737 TRACE [UNICAST] (Timer-4,saudade-16366) saudade-16366 --> XMIT(saudade-59431: #6) 2011-06-09 13:43:59,738 TRACE [UDP] (Timer-4,saudade-16366) sending msg to saudade-59431, src=saudade-16366, headers are RequestCorrelator: id=201, type=REQ, id=3, rsp_expected=true, UNICAST: DATA, seqno=6, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:43:59,739 TRACE [UDP] (OOB-10,saudade-16366) received [dst: saudade-16366, src: saudade-59431 (3 headers), size=4 bytes, flags=OOB|DONT_BUNDLE|NO_FC], headers are RequestCorrelator: id=201, type=RSP, id=3, rsp_expected=false, UNICAST: DATA, seqno=5, conn_id=1, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:44:00,938 TRACE [UNICAST] (Timer-5,saudade-16366) saudade-16366 --> XMIT(saudade-59431: #6) 2011-06-09 13:44:00,939 TRACE [UDP] (Timer-5,saudade-16366) sending msg to saudade-59431, src=saudade-16366, headers are RequestCorrelator: id=201, type=REQ, id=3, rsp_expected=true, UNICAST: DATA, seqno=6, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:44:00,942 TRACE [UDP] (OOB-11,saudade-16366) received [dst: saudade-16366, src: saudade-59431 (3 headers), size=4 bytes, flags=OOB|DONT_BUNDLE|NO_FC], headers are RequestCorrelator: id=201, type=RSP, id=3, rsp_expected=false, UNICAST: DATA, seqno=5, conn_id=1, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:44:00,987 TRACE [UDP] (OOB-6,saudade-16366) received [dst: , src: saudade-59431 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:44:01,072 TRACE [UDP] (Timer-2,saudade-16366) sending msg to null, src=saudade-16366, headers are FD_ALL: heartbeat, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:44:01,073 TRACE [UDP] (Timer-2,saudade-16366) looping back message [dst: , src: saudade-16366 (2 headers), size=0 bytes, flags=OOB] 2011-06-09 13:44:01,074 TRACE [UDP] (OOB-6,saudade-16366) received [dst: , src: saudade-16366 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:44:01,074 TRACE [UDP] (OOB-6,saudade-16366) received [dst: , src: saudade-16366 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:44:01,729 TRACE [UDP] (OOB-6,saudade-16366) received [dst: , src: saudade-59431 (2 headers), size=0 bytes, flags=OOB], headers are PING: [PING: type=GET_MBRS_REQ, cluster=Infinispan-Cluster, arg=own_addr=saudade-59431, view id=null, is_server=false, is_coord=false, logical_name=saudade-59431, physical_addrs=127.0.0.1:55358], UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:44:01,730 TRACE [PING] (OOB-6,saudade-16366) received GET_MBRS_REQ from saudade-59431, sending response [PING: type=GET_MBRS_RSP, arg=own_addr=saudade-16366, view id=[saudade-59431|1], is_server=true, is_coord=false, logical_name=saudade-16366] 2011-06-09 13:44:01,730 TRACE [UDP] (OOB-6,saudade-16366) sending msg to saudade-59431, src=saudade-16366, headers are PING: [PING: type=GET_MBRS_RSP, arg=own_addr=saudade-16366, view id=[saudade-59431|1], is_server=true, is_coord=false, logical_name=saudade-16366], UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:44:02,140 TRACE [UNICAST] (Timer-4,saudade-16366) saudade-16366 --> XMIT(saudade-59431: #6) 2011-06-09 13:44:02,140 TRACE [UDP] (Timer-4,saudade-16366) sending msg to saudade-59431, src=saudade-16366, headers are RequestCorrelator: id=201, type=REQ, id=3, rsp_expected=true, UNICAST: DATA, seqno=6, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:44:02,143 TRACE [UDP] (OOB-6,saudade-16366) received [dst: saudade-16366, src: saudade-59431 (3 headers), size=4 bytes, flags=OOB|DONT_BUNDLE|NO_FC], headers are RequestCorrelator: id=201, type=RSP, id=3, rsp_expected=false, UNICAST: DATA, seqno=5, conn_id=1, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:44:03,270 TRACE [UDP] (OOB-12,saudade-16366) received [dst: , src: saudade-59431 (2 headers), size=0 bytes, flags=OOB], headers are PING: [PING: type=GET_MBRS_REQ, cluster=Infinispan-Cluster, arg=own_addr=saudade-59431, view id=null, is_server=false, is_coord=false, logical_name=saudade-59431, physical_addrs=127.0.0.1:55358], UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:44:03,270 TRACE [PING] (OOB-12,saudade-16366) received GET_MBRS_REQ from saudade-59431, sending response [PING: type=GET_MBRS_RSP, arg=own_addr=saudade-16366, view id=[saudade-59431|1], is_server=true, is_coord=false, logical_name=saudade-16366] 2011-06-09 13:44:03,271 TRACE [UDP] (OOB-12,saudade-16366) sending msg to saudade-59431, src=saudade-16366, headers are PING: [PING: type=GET_MBRS_RSP, arg=own_addr=saudade-16366, view id=[saudade-59431|1], is_server=true, is_coord=false, logical_name=saudade-16366], UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:44:03,341 TRACE [UNICAST] (Timer-5,saudade-16366) saudade-16366 --> XMIT(saudade-59431: #6) 2011-06-09 13:44:03,342 TRACE [UDP] (Timer-5,saudade-16366) sending msg to saudade-59431, src=saudade-16366, headers are RequestCorrelator: id=201, type=REQ, id=3, rsp_expected=true, UNICAST: DATA, seqno=6, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:44:03,343 TRACE [UDP] (OOB-12,saudade-16366) received [dst: saudade-16366, src: saudade-59431 (3 headers), size=4 bytes, flags=OOB|DONT_BUNDLE|NO_FC], headers are RequestCorrelator: id=201, type=RSP, id=3, rsp_expected=false, UNICAST: DATA, seqno=5, conn_id=1, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:44:03,988 TRACE [UDP] (OOB-13,saudade-16366) received [dst: , src: saudade-59431 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:44:04,074 TRACE [UDP] (Timer-2,saudade-16366) sending msg to null, src=saudade-16366, headers are FD_ALL: heartbeat, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:44:04,074 TRACE [UDP] (Timer-2,saudade-16366) looping back message [dst: , src: saudade-16366 (2 headers), size=0 bytes, flags=OOB] 2011-06-09 13:44:04,074 TRACE [UDP] (OOB-13,saudade-16366) received [dst: , src: saudade-16366 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:44:04,076 TRACE [UDP] (OOB-14,saudade-16366) received [dst: , src: saudade-16366 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:44:04,542 TRACE [UNICAST] (Timer-4,saudade-16366) saudade-16366 --> XMIT(saudade-59431: #6) 2011-06-09 13:44:04,543 TRACE [UDP] (Timer-4,saudade-16366) sending msg to saudade-59431, src=saudade-16366, headers are RequestCorrelator: id=201, type=REQ, id=3, rsp_expected=true, UNICAST: DATA, seqno=6, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:44:04,545 TRACE [UDP] (OOB-14,saudade-16366) received [dst: saudade-16366, src: saudade-59431 (3 headers), size=4 bytes, flags=OOB|DONT_BUNDLE|NO_FC], headers are RequestCorrelator: id=201, type=RSP, id=3, rsp_expected=false, UNICAST: DATA, seqno=5, conn_id=1, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:44:05,743 TRACE [UNICAST] (Timer-5,saudade-16366) saudade-16366 --> XMIT(saudade-59431: #6) 2011-06-09 13:44:05,744 TRACE [UDP] (Timer-5,saudade-16366) sending msg to saudade-59431, src=saudade-16366, headers are RequestCorrelator: id=201, type=REQ, id=3, rsp_expected=true, UNICAST: DATA, seqno=6, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:44:05,745 TRACE [UDP] (OOB-13,saudade-16366) received [dst: saudade-16366, src: saudade-59431 (3 headers), size=4 bytes, flags=OOB|DONT_BUNDLE|NO_FC], headers are RequestCorrelator: id=201, type=RSP, id=3, rsp_expected=false, UNICAST: DATA, seqno=5, conn_id=1, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:44:06,945 TRACE [UNICAST] (Timer-2,saudade-16366) saudade-16366 --> XMIT(saudade-59431: #6) 2011-06-09 13:44:06,946 TRACE [UDP] (Timer-2,saudade-16366) sending msg to saudade-59431, src=saudade-16366, headers are RequestCorrelator: id=201, type=REQ, id=3, rsp_expected=true, UNICAST: DATA, seqno=6, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:44:06,948 TRACE [UDP] (OOB-15,saudade-16366) received [dst: saudade-16366, src: saudade-59431 (3 headers), size=4 bytes, flags=OOB|DONT_BUNDLE|NO_FC], headers are RequestCorrelator: id=201, type=RSP, id=3, rsp_expected=false, UNICAST: DATA, seqno=5, conn_id=1, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:44:06,988 TRACE [UDP] (OOB-16,saudade-16366) received [dst: , src: saudade-59431 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:44:07,091 TRACE [UDP] (Timer-3,saudade-16366) sending msg to null, src=saudade-16366, headers are FD_ALL: heartbeat, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:44:07,091 TRACE [UDP] (Timer-3,saudade-16366) looping back message [dst: , src: saudade-16366 (2 headers), size=0 bytes, flags=OOB] 2011-06-09 13:44:07,097 TRACE [UDP] (OOB-16,saudade-16366) received [dst: , src: saudade-16366 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:44:07,102 TRACE [UDP] (OOB-16,saudade-16366) received [dst: , src: saudade-16366 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:44:07,626 TRACE [STABLE] (main) saudade-16366: resetting digest from NAKACK: [saudade-59431#4, saudade-16366#0] 2011-06-09 13:44:07,627 DEBUG [STABLE] (main) resuming message garbage collection 2011-06-09 13:44:07,635 TRACE [RpcManagerImpl] (Incoming-2,saudade-16366) replication exception: org.infinispan.util.concurrent.TimeoutException: Timed out after 10 seconds waiting for a response from saudade-59431 at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$ReplicationTask.call(CommandAwareRpcDispatcher.java:267) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:116) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:436) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:131) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:155) at org.infinispan.statetransfer.StateTransferManagerImpl.mimicPartialFlushViaRPC(StateTransferManagerImpl.java:301) at org.infinispan.statetransfer.StateTransferManagerImpl.applyTransactionLog(StateTransferManagerImpl.java:254) at org.infinispan.statetransfer.StateTransferManagerImpl.applyState(StateTransferManagerImpl.java:322) at org.infinispan.remoting.InboundInvocationHandlerImpl.applyState(InboundInvocationHandlerImpl.java:230) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.setState(JGroupsTransport.java:602) at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUpEvent(MessageDispatcher.java:711) at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:771) at org.jgroups.JChannel.up(JChannel.java:1441) at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1074) at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.connectToStateProvider(STREAMING_STATE_TRANSFER.java:523) at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.handleStateRsp(STREAMING_STATE_TRANSFER.java:462) at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.up(STREAMING_STATE_TRANSFER.java:223) at org.jgroups.protocols.FRAG2.up(FRAG2.java:189) at org.jgroups.protocols.FlowControl.up(FlowControl.java:418) at org.jgroups.protocols.FlowControl.up(FlowControl.java:400) at org.jgroups.protocols.pbcast.GMS.up(GMS.java:891) at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:246) at org.jgroups.protocols.UNICAST.handleDataReceived(UNICAST.java:613) at org.jgroups.protocols.UNICAST.up(UNICAST.java:294) at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:703) at org.jgroups.protocols.BARRIER.up(BARRIER.java:119) at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:177) at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:275) at org.jgroups.protocols.MERGE2.up(MERGE2.java:209) at org.jgroups.protocols.Discovery.up(Discovery.java:291) at org.jgroups.protocols.PING.up(PING.java:66) at org.jgroups.protocols.TP.passMessageUp(TP.java:1102) at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1658) at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1640) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:637) 2011-06-09 13:44:07,637 TRACE [JBossMarshaller] (Incoming-2,saudade-16366) Stop unmarshaller 2011-06-09 13:44:07,637 ERROR [JGroupsTransport] (Incoming-2,saudade-16366) ISPN00096: Caught while requesting or applying state org.infinispan.statetransfer.StateTransferException: org.infinispan.util.concurrent.TimeoutException: Timed out after 10 seconds waiting for a response from saudade-59431 at org.infinispan.statetransfer.StateTransferManagerImpl.applyState(StateTransferManagerImpl.java:333) at org.infinispan.remoting.InboundInvocationHandlerImpl.applyState(InboundInvocationHandlerImpl.java:230) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.setState(JGroupsTransport.java:602) at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUpEvent(MessageDispatcher.java:711) at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:771) at org.jgroups.JChannel.up(JChannel.java:1441) at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1074) at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.connectToStateProvider(STREAMING_STATE_TRANSFER.java:523) at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.handleStateRsp(STREAMING_STATE_TRANSFER.java:462) at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.up(STREAMING_STATE_TRANSFER.java:223) at org.jgroups.protocols.FRAG2.up(FRAG2.java:189) at org.jgroups.protocols.FlowControl.up(FlowControl.java:418) at org.jgroups.protocols.FlowControl.up(FlowControl.java:400) at org.jgroups.protocols.pbcast.GMS.up(GMS.java:891) at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:246) at org.jgroups.protocols.UNICAST.handleDataReceived(UNICAST.java:613) at org.jgroups.protocols.UNICAST.up(UNICAST.java:294) at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:703) at org.jgroups.protocols.BARRIER.up(BARRIER.java:119) at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:177) at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:275) at org.jgroups.protocols.MERGE2.up(MERGE2.java:209) at org.jgroups.protocols.Discovery.up(Discovery.java:291) at org.jgroups.protocols.PING.up(PING.java:66) at org.jgroups.protocols.TP.passMessageUp(TP.java:1102) at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1658) at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1640) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:637) Caused by: org.infinispan.util.concurrent.TimeoutException: Timed out after 10 seconds waiting for a response from saudade-59431 at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$ReplicationTask.call(CommandAwareRpcDispatcher.java:267) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:116) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:436) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:131) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:155) at org.infinispan.statetransfer.StateTransferManagerImpl.mimicPartialFlushViaRPC(StateTransferManagerImpl.java:301) at org.infinispan.statetransfer.StateTransferManagerImpl.applyTransactionLog(StateTransferManagerImpl.java:254) at org.infinispan.statetransfer.StateTransferManagerImpl.applyState(StateTransferManagerImpl.java:322) ... 29 more 2011-06-09 13:44:07,637 WARN [RpcManagerImpl] (main) ISPN00075: Could not find available peer for state, backing off and retrying 2011-06-09 13:44:07,638 DEBUG [STREAMING_STATE_TRANSFER] (Incoming-2,saudade-16366) State reader is closing the socket 2011-06-09 13:44:07,639 DEBUG [STREAMING_STATE_TRANSFER] (Incoming-2,saudade-16366) STATE_TRANSFER_INPUTSTREAM_CLOSED received 2011-06-09 13:44:07,639 TRACE [STABLE] (Incoming-2,saudade-16366) saudade-16366: resetting digest from NAKACK: [saudade-59431#4, saudade-16366#0] 2011-06-09 13:44:07,639 DEBUG [STABLE] (Incoming-2,saudade-16366) resuming message garbage collection 2011-06-09 13:44:07,639 TRACE [BARRIER] (Incoming-2,saudade-16366) barrier was opened 2011-06-09 13:44:07,639 TRACE [UNICAST] (OOB-7,saudade-16366) saudade-16366 <-- DATA(saudade-59431: #5, conn_id=1) 2011-06-09 13:44:07,639 TRACE [UNICAST] (Incoming-1,saudade-16366) saudade-16366 <-- ACK(saudade-59431: #6) 2011-06-09 13:44:07,640 TRACE [UNICAST] (OOB-9,saudade-16366) saudade-16366 <-- DATA(saudade-59431: #5, conn_id=1) 2011-06-09 13:44:07,640 TRACE [UNICAST] (OOB-9,saudade-16366) saudade-16366 --> ACK(saudade-59431: #5) 2011-06-09 13:44:07,640 TRACE [UNICAST] (OOB-11,saudade-16366) saudade-16366 <-- DATA(saudade-59431: #5, conn_id=1) 2011-06-09 13:44:07,640 TRACE [UNICAST] (OOB-8,saudade-16366) saudade-16366 <-- DATA(saudade-59431: #5, conn_id=1) 2011-06-09 13:44:07,640 TRACE [UNICAST] (OOB-12,saudade-16366) saudade-16366 <-- DATA(saudade-59431: #5, conn_id=1) 2011-06-09 13:44:07,639 TRACE [JBossMarshaller] (OOB-7,saudade-16366) Start unmarshaller after retrieving marshaller from thread local 2011-06-09 13:44:07,640 TRACE [UNICAST] (OOB-15,saudade-16366) saudade-16366 <-- DATA(saudade-59431: #5, conn_id=1) 2011-06-09 13:44:07,640 TRACE [UNICAST] (OOB-14,saudade-16366) saudade-16366 <-- DATA(saudade-59431: #5, conn_id=1) 2011-06-09 13:44:07,640 TRACE [UNICAST] (OOB-6,saudade-16366) saudade-16366 <-- DATA(saudade-59431: #5, conn_id=1) 2011-06-09 13:44:07,640 TRACE [UDP] (OOB-9,saudade-16366) sending msg to saudade-59431, src=saudade-16366, headers are UNICAST: ACK, seqno=5, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:44:07,640 TRACE [UNICAST] (OOB-10,saudade-16366) saudade-16366 <-- DATA(saudade-59431: #5, conn_id=1) 2011-06-09 13:44:07,640 TRACE [VersionAwareMarshaller] (OOB-7,saudade-16366) Read version 500 2011-06-09 13:44:07,641 TRACE [JBossMarshaller] (OOB-7,saudade-16366) Stop unmarshaller 2011-06-09 13:44:07,641 TRACE [Request] (OOB-7,saudade-16366) received response for request 3, sender=saudade-59431, val=null 2011-06-09 13:44:07,640 TRACE [UNICAST] (OOB-13,saudade-16366) saudade-16366 <-- DATA(saudade-59431: #5, conn_id=1) 2011-06-09 13:44:09,639 INFO [RpcManagerImpl] (main) ISPN00074: Trying to fetch state from saudade-59431 2011-06-09 13:44:09,639 DEBUG [STREAMING_STATE_TRANSFER] (main) GET_STATE: asking saudade-59431 for state, passing down a SUSPEND_STABLE event, timeout=10000 2011-06-09 13:44:09,639 DEBUG [STABLE] (main) suspending message garbage collection 2011-06-09 13:44:09,639 DEBUG [STABLE] (main) resume task started, max_suspend_time=11000 2011-06-09 13:44:09,639 TRACE [UNICAST] (main) saudade-16366 --> DATA(saudade-59431: #7, conn_id=0) 2011-06-09 13:44:09,639 TRACE [UDP] (main) sending msg to saudade-59431, src=saudade-16366, headers are STREAMING_STATE_TRANSFER: type=STATE_REQ, sender=saudade-16366 id=0, UNICAST: DATA, seqno=7, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:44:09,644 TRACE [UDP] (Incoming-2,saudade-16366) received [dst: saudade-16366, src: saudade-59431 (3 headers), size=0 bytes], headers are STREAMING_STATE_TRANSFER: type=STATE_RSP, sender=saudade-59431 id=0, digest=saudade-16366: [0 : 0 (0)], saudade-59431: [0 : 4 (4)], UNICAST: DATA, seqno=6, conn_id=1, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:44:09,644 TRACE [UNICAST] (Incoming-2,saudade-16366) saudade-16366 <-- DATA(saudade-59431: #6, conn_id=1) 2011-06-09 13:44:09,644 TRACE [UNICAST] (Incoming-2,saudade-16366) saudade-16366 --> ACK(saudade-59431: #6) 2011-06-09 13:44:09,644 TRACE [UDP] (Incoming-2,saudade-16366) sending msg to saudade-59431, src=saudade-16366, headers are UNICAST: ACK, seqno=6, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:44:09,645 TRACE [BARRIER] (Incoming-2,saudade-16366) barrier was closed 2011-06-09 13:44:09,645 DEBUG [NAKACK] (Incoming-2,saudade-16366) [overwriteDigest()] existing digest: saudade-16366: [0 : 0 (0)], saudade-59431: [0 : 4 (4)] new digest: saudade-16366: [0 : 0 (0)], saudade-59431: [0 : 4 (4)] resulting digest: saudade-16366: [0 : 0 (0)], saudade-59431: [0 : 4 (4)] 2011-06-09 13:44:09,645 DEBUG [STREAMING_STATE_TRANSFER] (Incoming-2,saudade-16366) Connecting to state provider /127.0.0.1:50602, original buffer size was 262140 and was reset to 8192 2011-06-09 13:44:09,646 DEBUG [STREAMING_STATE_TRANSFER] (Incoming-2,saudade-16366) Connected to state provider, my end of the socket is /127.0.0.1:50608 passing inputstream up... 2011-06-09 13:44:09,646 TRACE [JGroupsTransport] (Incoming-2,saudade-16366) Received state for cache named 'test'. Attempting to apply state. 2011-06-09 13:44:09,647 DEBUG [StateTransferManagerImpl] (Incoming-2,saudade-16366) Applying state 2011-06-09 13:44:09,647 TRACE [JBossMarshaller] (Incoming-2,saudade-16366) Start unmarshaller after retrieving marshaller from thread local 2011-06-09 13:44:09,658 TRACE [UDP] (Incoming-1,saudade-16366) received [dst: saudade-16366, src: saudade-59431 (2 headers), size=0 bytes], headers are UNICAST: ACK, seqno=7, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:44:09,661 TRACE [VersionAwareMarshaller] (Incoming-2,saudade-16366) Read version 500 2011-06-09 13:44:09,661 TRACE [StateTransferManagerImpl] (Incoming-2,saudade-16366) Integrating transaction log 2011-06-09 13:44:09,661 TRACE [StateTransferManagerImpl] (Incoming-2,saudade-16366) Applying commit log 2011-06-09 13:44:09,661 TRACE [StateTransferManagerImpl] (Incoming-2,saudade-16366) Finished applying commit log 2011-06-09 13:44:09,662 TRACE [JGroupsTransport] (Incoming-2,saudade-16366) dests=[saudade-59431], command=StateTransferControlCommand{enabled=true}, mode=SYNCHRONOUS, timeout=10000 2011-06-09 13:44:09,663 TRACE [CommandAwareRpcDispatcher] (Incoming-2,saudade-16366) Replication task sending StateTransferControlCommand{enabled=true} to addresses [saudade-59431] 2011-06-09 13:44:09,663 TRACE [JBossMarshaller] (Incoming-2,saudade-16366) Start marshaller after retrieving marshaller from thread local 2011-06-09 13:44:09,663 TRACE [VersionAwareMarshaller] (Incoming-2,saudade-16366) Wrote version 500 2011-06-09 13:44:09,663 TRACE [JBossMarshaller] (Incoming-2,saudade-16366) Stop marshaller 2011-06-09 13:44:09,663 TRACE [Request] (Incoming-2,saudade-16366) sending request (id=4) 2011-06-09 13:44:09,665 TRACE [UNICAST] (Incoming-2,saudade-16366) saudade-16366 --> DATA(saudade-59431: #8, conn_id=0) 2011-06-09 13:44:09,667 TRACE [UDP] (Incoming-2,saudade-16366) sending msg to saudade-59431, src=saudade-16366, headers are RequestCorrelator: id=201, type=REQ, id=4, rsp_expected=true, UNICAST: DATA, seqno=8, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:44:09,671 TRACE [UDP] (OOB-13,saudade-16366) received [dst: saudade-16366, src: saudade-59431 (3 headers), size=4 bytes, flags=OOB|DONT_BUNDLE|NO_FC], headers are RequestCorrelator: id=201, type=RSP, id=4, rsp_expected=false, UNICAST: DATA, seqno=7, conn_id=1, UDP: [channel_name=Infinispan-Cluster] 2011-06-09 13:44:09,672 TRACE [UDP] (Incoming-3,saudade-16366) received [dst: saudade-16366, src: saudade-59431 (2 headers), size=0 bytes], headers are UNICAST: ACK, seqno=8, UDP: [channel_name=Infinispan-Cluster]