Uploaded image for project: 'Infinispan'
  1. Infinispan
  2. ISPN-5142

Cross site state transfer - retry mechanism not working correctly

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Major
    • 7.1.0.CR2
    • 7.1.0.Beta1
    • State Transfer
    • None

    Description

      2 sites - main (2 nodes), backup (3 nodes)

      Scenario:
      1. take consumer site offline
      2. write data into producer site
      3. invoke push(backup) operation
      4. write data into consumer site (such that key set overlaps with the one used in step#2)

      Configuration:

      • sync backup, 2pc
      • Pessimistic TX, multiple operations in a single TX (e.g. 20)

      Consumer site logs indicate occuring lock acquisition problems, leading to chunk not being applied. In that case, the chunk should be re-sent, however, the retry mechanism does not wait for the reply from the retry.

      pruivo's note: retry mechanism can end up in an infinite loop

      11:25:53,549 WARN [org.infinispan.xsite.statetransfer.XSiteStateConsumerImpl] (remote-thread-3) ISPN000291: Unable to apply X-Site state chunk.
      org.infinispan.util.concurrent.TimeoutException: Unable to acquire lock after [3 seconds] on key [key_0000000000002652] for requestor [GlobalTransaction:<localhost-17627>:25:local]! Lock held by [GlobalTransaction:<localhost-11416>:43:remote]
      at org.infinispan.util.concurrent.locks.LockManagerImpl.lock(LockManagerImpl.java:198)
      at org.infinispan.util.concurrent.locks.LockManagerImpl.acquireLock(LockManagerImpl.java:171)
      at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockKeyAndCheckOwnership(AbstractTxLockingInterceptor.java:169)
      at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.lockAndRegisterBackupLock(PessimisticLockingInterceptor.java:304)
      at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitPutKeyValueCommand(PessimisticLockingInterceptor.java:101)
      at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
      at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)
      at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:34)
      at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
      at org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:344)
      at org.infinispan.interceptors.TxInterceptor.visitPutKeyValueCommand(TxInterceptor.java:241)
      at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
      at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)
      at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:34)
      at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
      at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:183)
      at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:119)
      at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
      at org.infinispan.interceptors.CacheMgmtInterceptor.updateStoreStatistics(CacheMgmtInterceptor.java:148)
      at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:134)
      at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
      at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:104)
      at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:73)
      at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:34)
      at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
      at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333)
      at org.infinispan.xsite.statetransfer.XSiteStateConsumerImpl.applyStateInTransaction(XSiteStateConsumerImpl.java:111)
      at org.infinispan.xsite.statetransfer.XSiteStateConsumerImpl.applyState(XSiteStateConsumerImpl.java:92)
      at org.infinispan.xsite.statetransfer.XSiteStatePushCommand.perform(XSiteStatePushCommand.java:50)
      at org.infinispan.remoting.LocalInvocation.call(LocalInvocation.java:43)
      at org.infinispan.xsite.BackupReceiverImpl.handleStateTransferState(BackupReceiverImpl.java:140)
      at org.infinispan.xsite.statetransfer.XSiteStatePushCommand.performInLocalSite(XSiteStatePushCommand.java:32)
      at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$3.run(CommandAwareRpcDispatcher.java:250)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:745)

      Attachments

        Activity

          Public project attachment banner

            context keys: [headless, issue, helper, isAsynchronousRequest, project, action, user]
            current Project key: ISPN

            People

              pruivo@redhat.com Pedro Ruivo
              mcimbora_jira Matej ─îimbora (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: