Loading...

XML

Word

Printable

Details

Type: Bug
Resolution: Done
Priority: Major
Fix Version/s: 8.1.0.Final
Affects Version/s: 6.0.0.Beta1
Component/s: Core
Labels:
- testsuite_stability

Bugzilla References:
https://bugzilla.redhat.com/show_bug.cgi?id=1016024

Description

The tests:

org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest.testAsync
org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest.testSync

fail randomly on RHEL 5,6 - 32 and 64 bit machines with JDK6.

The error message is:

2013-10-01 11:05:02,877 WARN  [CacheNotifierImpl] (testng-StaleLocksWithLockOnlyTxDuringStateTransferTest) ISPN000133: Attempted to register listener of class class org.infinispan.statetransfer.StateProviderImpl$$EnhancerByMockitoWithCGLIB$$1b069019, but no valid, public methods annotated with method-level event annotations found! Ignoring listener.
2013-10-01 11:05:05,306 WARN  [arjuna] (Transaction Reaper) ARJUNA012117: TransactionReaper::check timeout for TX 0:ffff0a1059e0:ce1c:524adeb6:2fd9 in state  RUN
2013-10-01 11:05:05,308 WARN  [jta] (Transaction Reaper Worker 0) ARJUNA016029: SynchronizationImple.afterCompletion - failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=[StaleLocksWithCommitDuringStateTransferTest-NodeK-42948, StaleLocksWithCommitDuringStateTransferTest-NodeL-43885], isMarkedForRollback=false, lockedKeys=null, backupKeyLocks=[MagicKey#k2{f9dc9ad7@StaleLocksWithCommitDuringStateTransferTest-NodeL-43885/39}], topologyId=2, isFromStateTransfer=false} org.infinispan.transaction.synchronization.SyncLocalTransaction@32a6} org.infinispan.transaction.synchronization.SynchronizationAdapter@32c5 with exception
org.infinispan.commons.CacheException: Could not commit.
	at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:67)
	at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.afterCompletion(SynchronizationImple.java:96)
	at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.afterCompletion(TwoPhaseCoordinator.java:402)
	at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.afterCompletion(TwoPhaseCoordinator.java:334)
	at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.cancel(TwoPhaseCoordinator.java:120)
	at com.arjuna.ats.arjuna.AtomicAction.cancel(AtomicAction.java:215)
	at com.arjuna.ats.arjuna.coordinator.TransactionReaper.doCancellations(TransactionReaper.java:370)
	at com.arjuna.ats.internal.arjuna.coordinator.ReaperWorkerThread.run(ReaperWorkerThread.java:78)
Caused by: javax.transaction.xa.XAException
	at org.infinispan.transaction.TransactionCoordinator.rollback(TransactionCoordinator.java:186)
	at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:65)
	... 7 more
2013-10-01 11:05:05,308 WARN  [arjuna] (Transaction Reaper Worker 0) ARJUNA012127: TwoPhaseCoordinator.afterCompletion - returned failure for SynchronizationImple< 0:ffff0a1059e0:ce1c:524adeb6:2fda, SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=[StaleLocksWithCommitDuringStateTransferTest-NodeK-42948, StaleLocksWithCommitDuringStateTransferTest-NodeL-43885], isMarkedForRollback=false, lockedKeys=null, backupKeyLocks=[MagicKey#k2{f9dc9ad7@StaleLocksWithCommitDuringStateTransferTest-NodeL-43885/39}], topologyId=2, isFromStateTransfer=false} org.infinispan.transaction.synchronization.SyncLocalTransaction@32a6} org.infinispan.transaction.synchronization.SynchronizationAdapter@32c5 >
2013-10-01 11:05:05,309 WARN  [arjuna] (Transaction Reaper Worker 0) ARJUNA012121: TransactionReaper::doCancellations worker Thread[Transaction Reaper Worker 0,5,main] successfully canceled TX 0:ffff0a1059e0:ce1c:524adeb6:2fd9
2013-10-01 11:05:12,943 ERROR [CheckPoint] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA) Timed out waiting for event allow_get_transactions_1_from_StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB-26402 * 1 (available = 0, total = 0
2013-10-01 11:05:12,943 ERROR [InboundInvocationHandlerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA) Exception executing command
java.util.concurrent.TimeoutException: Timed out waiting for event allow_get_transactions_1_from_StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB-26402
	at org.infinispan.test.fwk.CheckPoint.awaitStrict(CheckPoint.java:43)
	at org.infinispan.test.fwk.CheckPoint.awaitStrict(CheckPoint.java:33)
	at org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest$1.answer(StaleLocksWithLockOnlyTxDuringStateTransferTest.java:101)
	at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34)
	at org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:91)
	at org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29)
	at org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:38)
	at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:51)
	at org.infinispan.statetransfer.StateProviderImpl$$EnhancerByMockitoWithCGLIB$$1b069019.getTransactionsForSegments(<generated>)
	at org.infinispan.statetransfer.StateRequestCommand.perform(StateRequestCommand.java:65)
	at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:97)
	at org.infinispan.remoting.InboundInvocationHandlerImpl.access$000(InboundInvocationHandlerImpl.java:46)
	at org.infinispan.remoting.InboundInvocationHandlerImpl$2.run(InboundInvocationHandlerImpl.java:169)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:724)
2013-10-01 11:05:12,950 WARN  [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000209: Failed to retrieve transactions for segments [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 16, 19, 18, 21, 20, 23, 22, 25, 24, 27, 26, 29, 28, 31, 30, 34, 35, 32, 33, 38, 39, 36, 37, 42, 43, 40, 41, 46, 47, 44, 45, 51, 50, 49, 48, 55, 54, 53, 52, 59, 58, 57, 56] of cache testCache from node StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865 (node will not be retried)
org.infinispan.remoting.RemoteException: ISPN000217: Received exception from StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865, see cause for remote stack trace
	at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:38)
	at org.infinispan.remoting.transport.AbstractTransport.parseResponseAndAddToResponseList(AbstractTransport.java:63)
	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:534)
	at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:288)
	at org.infinispan.statetransfer.StateConsumerImpl.getTransactions(StateConsumerImpl.java:686)
	at org.infinispan.statetransfer.StateConsumerImpl.requestTransactions(StateConsumerImpl.java:653)
	at org.infinispan.statetransfer.StateConsumerImpl.addTransfers(StateConsumerImpl.java:603)
	at org.infinispan.statetransfer.StateConsumerImpl.onTopologyUpdate(StateConsumerImpl.java:367)
	at org.infinispan.statetransfer.StateTransferManagerImpl.doTopologyUpdate(StateTransferManagerImpl.java:178)
	at org.infinispan.statetransfer.StateTransferManagerImpl.access$000(StateTransferManagerImpl.java:38)
	at org.infinispan.statetransfer.StateTransferManagerImpl$1.rebalance(StateTransferManagerImpl.java:105)
	at org.infinispan.topology.LocalTopologyManagerImpl.handleRebalance(LocalTopologyManagerImpl.java:228)
	at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:155)
	at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:124)
	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$4.run(CommandAwareRpcDispatcher.java:270)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:724)
Caused by: java.util.concurrent.TimeoutException: Timed out waiting for event allow_get_transactions_1_from_StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB-26402
	at org.infinispan.test.fwk.CheckPoint.awaitStrict(CheckPoint.java:43)
	at org.infinispan.test.fwk.CheckPoint.awaitStrict(CheckPoint.java:33)
	at org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest$1.answer(StaleLocksWithLockOnlyTxDuringStateTransferTest.java:101)
	at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34)
	at org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:91)
	at org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29)
	at org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:38)
	at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:51)
	at org.infinispan.statetransfer.StateProviderImpl$$EnhancerByMockitoWithCGLIB$$1b069019.getTransactionsForSegments(<generated>)
	at org.infinispan.statetransfer.StateRequestCommand.perform(StateRequestCommand.java:65)
	at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:97)
	at org.infinispan.remoting.InboundInvocationHandlerImpl.access$000(InboundInvocationHandlerImpl.java:46)
	at org.infinispan.remoting.InboundInvocationHandlerImpl$2.run(InboundInvocationHandlerImpl.java:169)
	... 3 more
2013-10-01 11:05:12,951 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 0 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:12,951 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 1 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:12,951 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 2 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:12,951 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 3 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:12,952 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 4 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:12,952 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 5 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:12,952 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 6 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:12,952 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 7 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:12,952 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 8 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:12,952 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 9 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:12,952 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 10 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:12,952 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 11 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:12,953 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 12 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:12,953 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 13 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:12,953 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 14 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:12,954 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 15 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:12,954 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 17 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:12,954 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 16 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:12,955 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 19 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:12,955 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 18 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:12,955 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 21 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:12,955 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 20 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:12,956 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 23 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:12,956 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 22 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:12,956 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 25 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:12,956 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 24 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:12,957 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 27 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:12,957 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 26 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:12,957 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 29 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:12,957 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 28 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:12,958 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 31 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:12,958 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 30 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:12,958 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 34 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:12,959 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 35 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:12,959 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 32 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:12,959 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 33 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:12,959 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 38 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:12,960 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 39 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:12,960 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 36 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:12,960 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 37 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:12,960 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 42 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:12,961 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 43 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:12,961 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 40 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:12,961 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 41 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:12,962 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 46 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:12,962 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 47 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:12,999 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 44 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,000 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 45 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,000 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 51 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,001 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 50 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,001 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 49 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,001 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 48 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,002 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 55 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,002 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 54 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,002 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 53 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,002 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 52 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,003 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 59 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,003 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 58 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,003 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 57 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,004 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 56 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,004 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 0 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,004 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 1 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,005 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 2 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,005 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 3 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,005 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 4 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,005 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 5 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,006 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 6 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,006 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 7 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,006 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 8 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,007 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 9 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,007 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 10 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,007 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 11 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,008 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 12 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,008 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 13 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,008 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 14 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,008 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 15 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,009 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 17 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,009 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 16 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,009 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 19 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,010 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 18 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,010 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 21 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,010 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 20 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,010 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 23 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,011 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 22 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,011 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 25 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,011 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 24 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,012 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 27 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,012 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 26 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,012 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 29 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,012 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 28 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,013 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 31 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,013 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 30 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,013 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 34 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,014 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 35 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,014 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 32 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,014 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 33 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,014 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 38 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,015 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 39 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,015 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 36 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,015 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 37 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,016 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 42 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,016 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 43 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,016 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 40 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,016 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 41 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,017 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 46 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,017 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 47 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,017 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 44 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,018 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 45 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,018 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 51 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,018 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 50 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,019 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 49 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,019 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 48 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,019 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 55 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,019 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 54 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,020 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 53 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,020 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 52 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,020 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 59 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,021 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 58 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,021 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 57 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
2013-10-01 11:05:13,021 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 56 of cache testCache. Current owners are:  [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
[testng-StaleLocksWithLockOnlyTxDuringStateTransferTest] Test testAsync(org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest) failed.
2013-10-01 11:05:13,035 ERROR [UnitTestTestNGListener] (testng-StaleLocksWithLockOnlyTxDuringStateTransferTest) Test testAsync(org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest) failed.
java.lang.IllegalStateException: Thread already timed out waiting for event allow_get_transactions_1_from_StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB-26402
	at org.infinispan.test.fwk.CheckPoint.trigger(CheckPoint.java:131)
	at org.infinispan.test.fwk.CheckPoint.trigger(CheckPoint.java:116)
	at org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest.doTest(StaleLocksWithLockOnlyTxDuringStateTransferTest.java:144)
	at org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest.testAsync(StaleLocksWithLockOnlyTxDuringStateTransferTest.java:75)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:80)
	at org.testng.internal.Invoker.invokeMethod(Invoker.java:715)
	at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:907)
	at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1237)
	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
	at org.testng.TestRunner.privateRun(TestRunner.java:767)
	at org.testng.TestRunner.run(TestRunner.java:617)
	at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
	at org.testng.SuiteRunner.access$000(SuiteRunner.java:37)
	at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:368)
	at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:724)
Test suite progress: tests succeeded: 1536, failed: 1, skipped: 0.

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

StaleLocksWithLockOnlyTxDuringStateTransferTest.testSync.log.zip
179 kB
2013/10/15 4:01 AM
StaleLocksWithLockOnlyTxDuringStateTransferTest.testSync.log.zip
178 kB
2013/10/15 3:58 AM

Activity

People

Assignee:: Unassigned

Reporter:: Anna Manukyan

Votes:: 0 Vote for this issue

Watchers:: 6 Start watching this issue

Dates

Created:: 2013/10/07 6:33 AM

Updated:: 2020/02/07 6:05 AM

Resolved:: 2016/03/31 9:37 AM