Uploaded image for project: 'WildFly'
  1. WildFly
  2. WFLY-7958

Could not write method invocation failure due to HeuristicMixedException

    XMLWordPrintable

Details

    Description

      Occured during topology change on server during scenarios:
      eap-7x-failover-ejb-ejbremote-jvmkill-repl-async
      eap-7x-failover-ejb-ejbremote-shutdown-repl-async
      eap-7x-failover-ejb-ejbremote-shutdown-repl-sync
      eap-7x-failover-ejb-ejbremote-undeploy-repl-sync

      before "Finished local rebalance" INFO message was logged (scenarios
      eap-7x-failover-ejb-ejbremote-jvmkill-repl-async and
      eap-7x-failover-ejb-ejbremote-undeploy-repl-sync)
      or directly after "Received new cluster view" INFO message (scenario
      eap-7x-failover-ejb-ejbremote-shutdown-repl-async).

      Server log stacktrace:

      16:02:10,148 ERROR [org.jboss.as.ejb3.remote] (default task-111) WFLYEJB0150: Could not write method invocation failure for method public abstract int org.jboss.test.clusterbench.common.ejb.CommonStatefulSB.getSerialAndIncrement() on bean named RemoteStatefulSBImpl for appname clusterbench-ee7 modulename clusterbench-ee7-ejb distinctname  due to: org.infinispan.commons.CacheException: javax.transaction.HeuristicMixedException
      [JBossINF] 	at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:102)
      [JBossINF] 	at org.jboss.as.ejb3.cache.distributable.DistributableCache.getWeakAffinity(DistributableCache.java:69)
      [JBossINF] 	at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.getWeakAffinity(MethodInvocationMessageHandler.java:274)
      [JBossINF] 	at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.access$600(MethodInvocationMessageHandler.java:68)
      [JBossINF] 	at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler$1.run(MethodInvocationMessageHandler.java:244)
      [JBossINF] 	at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.processMessage(MethodInvocationMessageHandler.java:268)
      [JBossINF] 	at org.jboss.as.ejb3.remote.protocol.versionone.VersionOneProtocolChannelReceiver.processMessage(VersionOneProtocolChannelReceiver.java:213)
      [JBossINF] 	at org.jboss.as.ejb3.remote.protocol.versiontwo.VersionTwoProtocolChannelReceiver.processMessage(VersionTwoProtocolChannelReceiver.java:76)
      [JBossINF] 	at org.jboss.as.ejb3.remote.protocol.versionone.VersionOneProtocolChannelReceiver.handleMessage(VersionOneProtocolChannelReceiver.java:159)
      [JBossINF] 	at org.jboss.remoting3.remote.RemoteConnectionChannel.lambda$receiveMessage$2(RemoteConnectionChannel.java:365)
      [JBossINF] 	at org.jboss.remoting3.EndpointImpl$TrackingExecutor.lambda$execute$0(EndpointImpl.java:734)
      [JBossINF] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      [JBossINF] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      [JBossINF] 	at java.lang.Thread.run(Thread.java:745)
      [JBossINF] Caused by: javax.transaction.HeuristicMixedException
      [JBossINF] 	at org.infinispan.transaction.tm.DummyTransaction.finishResource(DummyTransaction.java:439)
      [JBossINF] 	at org.infinispan.transaction.tm.DummyTransaction.commitResources(DummyTransaction.java:448)
      [JBossINF] 	at org.infinispan.transaction.tm.DummyTransaction.runCommit(DummyTransaction.java:321)
      [JBossINF] 	at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:108)
      [JBossINF] 	at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:97)
      [JBossINF] 	... 13 more
      [JBossINF] Caused by: javax.transaction.xa.XAException
      [JBossINF] 	at org.infinispan.transaction.impl.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:207)
      [JBossINF] 	at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:159)
      [JBossINF] 	at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:114)
      [JBossINF] 	at org.infinispan.transaction.tm.DummyTransaction.finishResource(DummyTransaction.java:401)
      [JBossINF] 	... 17 more
      [JBossINF] Caused by: org.infinispan.util.concurrent.TimeoutException: Replication timeout for perf20
      [JBossINF] 	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801)
      [JBossINF] 	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$1(JGroupsTransport.java:642)
      [JBossINF] 	at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
      [JBossINF] 	at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
      [JBossINF] 	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
      [JBossINF] 	at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
      [JBossINF] 	at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:47)
      [JBossINF] 	at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:16)
      [JBossINF] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      [JBossINF] 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
      [JBossINF] 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
      [JBossINF] 	... 3 more
      

      Link to server log:
      http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/perflab_eap-7x-failover-ejb-ejbremote-shutdown-repl-async/5/console-perf20/

      Variant with HeuristicRollbackException logged on perf20 during scenario eap-7x-failover-ejb-ejbremote-jvmkill-repl-async (2 occurences):

      16:08:49,504 ERROR [org.jboss.as.ejb3.remote] (default task-70) WFLYEJB0150: Could not write method invocation failure for method public abstract int org.jboss.test.clusterbench.common.ejb.CommonStatefulSB.getSerialAndIncrement() on bean named RemoteStatefulSBImpl for appname clusterbench-ee7 modulename clusterbench-ee7-ejb distinctname  due to: org.infinispan.commons.CacheException: javax.transaction.HeuristicRollbackException
      [JBossINF] 	at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:102)
      [JBossINF] 	at org.jboss.as.ejb3.cache.distributable.DistributableCache.getWeakAffinity(DistributableCache.java:69)
      [JBossINF] 	at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.getWeakAffinity(MethodInvocationMessageHandler.java:274)
      [JBossINF] 	at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.access$600(MethodInvocationMessageHandler.java:68)
      [JBossINF] 	at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler$1.run(MethodInvocationMessageHandler.java:244)
      [JBossINF] 	at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.processMessage(MethodInvocationMessageHandler.java:268)
      [JBossINF] 	at org.jboss.as.ejb3.remote.protocol.versionone.VersionOneProtocolChannelReceiver.processMessage(VersionOneProtocolChannelReceiver.java:213)
      [JBossINF] 	at org.jboss.as.ejb3.remote.protocol.versiontwo.VersionTwoProtocolChannelReceiver.processMessage(VersionTwoProtocolChannelReceiver.java:76)
      [JBossINF] 	at org.jboss.as.ejb3.remote.protocol.versionone.VersionOneProtocolChannelReceiver.handleMessage(VersionOneProtocolChannelReceiver.java:159)
      [JBossINF] 	at org.jboss.remoting3.remote.RemoteConnectionChannel.lambda$receiveMessage$2(RemoteConnectionChannel.java:365)
      [JBossINF] 	at org.jboss.remoting3.EndpointImpl$TrackingExecutor.lambda$execute$0(EndpointImpl.java:734)
      [JBossINF] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      [JBossINF] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      [JBossINF] 	at java.lang.Thread.run(Thread.java:745)
      [JBossINF] Caused by: javax.transaction.HeuristicRollbackException
      [JBossINF] 	at org.infinispan.transaction.tm.DummyTransaction.finishResource(DummyTransaction.java:433)
      [JBossINF] 	at org.infinispan.transaction.tm.DummyTransaction.commitResources(DummyTransaction.java:448)
      [JBossINF] 	at org.infinispan.transaction.tm.DummyTransaction.runCommit(DummyTransaction.java:321)
      [JBossINF] 	at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:108)
      [JBossINF] 	at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:97)
      [JBossINF] 	... 13 more
      [JBossINF] Caused by: javax.transaction.xa.XAException
      [JBossINF] 	at org.infinispan.transaction.impl.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:213)
      [JBossINF] 	at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:159)
      [JBossINF] 	at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:114)
      [JBossINF] 	at org.infinispan.transaction.tm.DummyTransaction.finishResource(DummyTransaction.java:401)
      [JBossINF] 	... 17 more
      [JBossINF] Caused by: org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 27
      [JBossINF] 	at org.infinispan.statetransfer.StateTransferLockImpl.reportErrorAfterWait(StateTransferLockImpl.java:159)
      [JBossINF] 	at org.infinispan.statetransfer.StateTransferLockImpl.waitForTransactionData(StateTransferLockImpl.java:98)
      [JBossINF] 	at org.infinispan.interceptors.base.BaseStateTransferInterceptor.waitForTransactionData(BaseStateTransferInterceptor.java:97)
      [JBossINF] 	at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:249)
      [JBossINF] 	at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:255)
      [JBossINF] 	at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:87)
      [JBossINF] 	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
      [JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
      [JBossINF] 	at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:114)
      [JBossINF] 	at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:83)
      [JBossINF] 	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112)
      [JBossINF] 	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
      [JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
      [JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
      [JBossINF] 	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112)
      [JBossINF] 	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
      [JBossINF] 	at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:335)
      [JBossINF] 	at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:157)
      [JBossINF] 	... 19 more
      

      Link to server log:
      http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/perflab_eap-7x-failover-ejb-ejbremote-jvmkill-repl-async/2/console-perf20/

      Attachments

        Issue Links

          Activity

            People

              pferraro@redhat.com Paul Ferraro
              pferraro@redhat.com Paul Ferraro
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: