Uploaded image for project: 'JBoss Enterprise Application Platform'
  1. JBoss Enterprise Application Platform
  2. JBEAP-16662

WF transaction client returns XAER_RMERR during abort after server with subordinate transaction fails

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • 7.3.0.CD18
    • 7.0.9.GA, 7.1.6.GA, 7.2.0.GA, 7.3.0.CD15
    • Transactions
    • None
    • Hide
      1. get EAP
      2. get [crashrec TS|git@gitlab.mw.lab.eng.bos.redhat.com:jbossqe-eap/tests-transactions.git]
      3. export JBOSS_HOME=path-to-EAP
      4. get driver for PostgreSQL DB
      5. mvn clean verify -Dtest=TxPropagationJPACrashRecoveryTestCase#prepareHaltServer -Dds0.db=postgresql101 -Dds0.jdbc.driver.jar=postgresql-42.2.2.jar -Dds0.jdbc.driver.class=org.postgresql.Driver -Dds0.jdbc.xa-class=org.postgresql.xa.PGXADataSource -Dds0.jdbc.driver.path=path-to-driver -Dds0.hibernate.dialect=org.hibernate.dialect.PostgreSQL82Dialect -Dds0.db.name=crashrec -Dds0.user=crashrec -Dds0.pass=crashrec -Dds0.jdbc.url=jdbc:postgresql://postgres-101.rhev-ci-vms.eng.rdu2.redhat.com:5432/crashrec -Dds1.db=postgresql101 -Dds1.jdbc.driver.jar=postgresql-42.2.2.jar -Dds1.jdbc.driver.class=org.postgresql.Driver -Dds1.jdbc.xa-class=org.postgresql.xa.PGXADataSource -Dds1.jdbc.driver.path=path-to-driver -Dds1.hibernate.dialect=org.hibernate.dialect.PostgreSQL82Dialect -Dds1.db.name=crashrec2 -Dds1.user=crashrec2 -Dds1.pass=crashrec2 -Dds1.jdbc.url=jdbc:postgresql://postgres-101.rhev-ci-vms.eng.rdu2.redhat.com:5432/crashrec2 -DfailIfNoTests=false -Dts.jbossts1.node.identifier=TWkA -Dts.jbossts2.node.identifier=RJSN -Dts.jbossts3.node.identifier=WYYJ

      You can use your own DB

      Show
      get EAP get [crashrec TS|git@gitlab.mw.lab.eng.bos.redhat.com:jbossqe-eap/tests-transactions.git] export JBOSS_HOME=path-to-EAP get driver for PostgreSQL DB mvn clean verify -Dtest=TxPropagationJPACrashRecoveryTestCase#prepareHaltServer -Dds0.db=postgresql101 -Dds0.jdbc.driver.jar=postgresql-42.2.2.jar -Dds0.jdbc.driver.class=org.postgresql.Driver -Dds0.jdbc.xa-class=org.postgresql.xa.PGXADataSource -Dds0.jdbc.driver.path=path-to-driver -Dds0.hibernate.dialect=org.hibernate.dialect.PostgreSQL82Dialect -Dds0.db.name=crashrec -Dds0.user=crashrec -Dds0.pass=crashrec -Dds0.jdbc.url=jdbc:postgresql://postgres-101.rhev-ci-vms.eng.rdu2.redhat.com:5432/crashrec -Dds1.db=postgresql101 -Dds1.jdbc.driver.jar=postgresql-42.2.2.jar -Dds1.jdbc.driver.class=org.postgresql.Driver -Dds1.jdbc.xa-class=org.postgresql.xa.PGXADataSource -Dds1.jdbc.driver.path=path-to-driver -Dds1.hibernate.dialect=org.hibernate.dialect.PostgreSQL82Dialect -Dds1.db.name=crashrec2 -Dds1.user=crashrec2 -Dds1.pass=crashrec2 -Dds1.jdbc.url=jdbc:postgresql://postgres-101.rhev-ci-vms.eng.rdu2.redhat.com:5432/crashrec2 -DfailIfNoTests=false -Dts.jbossts1.node.identifier=TWkA -Dts.jbossts2.node.identifier=RJSN -Dts.jbossts3.node.identifier=WYYJ You can use your own DB

      Scenario

      1. Server1 enlists dummy XA resource and make JPA call
      2. Server1 calls a EJB on Server2
      3. Server2 enlists dummy XA resource and make JPA call
      4. Server2 is halted at the beginning of dummy XA resource
      5. Transaction is supposed to be rolled back

      We can see following lines in the log of Server1

      2019-04-02 21:22:41,707 TRACE [com.arjuna.ats.jta] (default task-2) XAResourceRecord.topLevelAbort for XAResourceRecord < resource:Subordinate XAResource at http-remoting://127.0.0.1:8180, txid:< formatId=131077, gtrid_length=32, bqual_length=36, tx_uid=0:ffff7f000001:-70ba203e:5ca3d272:2f, node_name=TWkA, branch_uid=0:ffff7f000001:-70ba203e:5ca3d272:35, subordinatenodename=null, eis_name=unknown eis name >, heuristic: TwoPhaseOutcome.FINISH_OK com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord@531d7d94 >, record id=0:ffff7f000001:-70ba203e:5ca3d272:36
      2019-04-02 21:22:41,708 WARN  [com.arjuna.ats.jta] (default task-2) ARJUNA016045: attempted rollback of < formatId=131077, gtrid_length=32, bqual_length=36, tx_uid=0:ffff7f000001:-70ba203e:5ca3d272:2f, node_name=TWkA, branch_uid=0:ffff7f000001:-70ba203e:5ca3d272:35, subordinatenodename=null, eis_name=unknown eis name > (Subordinate XAResource at http-remoting://127.0.0.1:8180) failed with exception code XAException.XAER_RMERR: javax.transaction.xa.XAException: WFTXN0026: Failed to send protocol message to remote peer
      	at org.wildfly.transaction.client.provider.remoting.TransactionClientChannel.rollback(TransactionClientChannel.java:98)
      	at org.wildfly.transaction.client.provider.remoting.RemotingRemoteTransactionPeer$1.rollback(RemotingRemoteTransactionPeer.java:149)
      	at org.wildfly.transaction.client.SubordinateXAResource.rollback(SubordinateXAResource.java:174)
      	at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelAbort(XAResourceRecord.java:362)
      	at com.arjuna.ats.arjuna.coordinator.BasicAction.doAbort(BasicAction.java:3023)
      	at com.arjuna.ats.arjuna.coordinator.BasicAction.doAbort(BasicAction.java:3002)
      	at com.arjuna.ats.arjuna.coordinator.BasicAction.phase2Abort(BasicAction.java:1981)
      	at com.arjuna.ats.arjuna.coordinator.BasicAction.End(BasicAction.java:1519)
      	at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:96)
      	at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
      	at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1288)
      	at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
      	at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89)
      	at org.wildfly.transaction.client.LocalTransaction.commitAndDissociate(LocalTransaction.java:75)
      	at org.wildfly.transaction.client.ContextTransactionManager.commit(ContextTransactionManager.java:71)
      	at org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:88)
      	at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:261)
      	at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:362)
      	at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:144)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
      	at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
      	at org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:81)
      	at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
      	at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
      	at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
      	at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
      	at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
      	at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
      	at org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:57)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
      	at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
      	at org.jboss.as.ejb3.component.interceptors.EjbExceptionTransformingInterceptorFactories$1.processInvocation(EjbExceptionTransformingInterceptorFactories.java:80)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
      	at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
      	at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
      	at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438)
      	at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:618)
      	at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
      	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
      	at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
      	at org.wildfly.security.auth.server.SecurityIdentity.runAsFunctionEx(SecurityIdentity.java:406)
      	at org.jboss.as.ejb3.remote.AssociationImpl.invokeWithIdentity(AssociationImpl.java:565)
      	at org.jboss.as.ejb3.remote.AssociationImpl.invokeMethod(AssociationImpl.java:546)
      	at org.jboss.as.ejb3.remote.AssociationImpl.lambda$receiveInvocationRequest$0(AssociationImpl.java:197)
      	at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
      	at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982)
      	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
      	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
      	at java.lang.Thread.run(Thread.java:748)
      Caused by: org.jboss.remoting3.NotOpenException: Writes closed
      	at org.jboss.remoting3.remote.RemoteConnectionChannel.openOutboundMessage(RemoteConnectionChannel.java:106)
      	at org.jboss.remoting3.remote.RemoteConnectionChannel.writeMessage(RemoteConnectionChannel.java:291)
      	at org.jboss.remoting3.util.MessageTracker.openMessage(MessageTracker.java:64)
      	at org.jboss.remoting3.util.InvocationTracker.allocateMessage(InvocationTracker.java:189)
      	at org.jboss.remoting3.util.InvocationTracker.allocateMessage(InvocationTracker.java:205)
      	at org.wildfly.transaction.client.provider.remoting.TransactionClientChannel.rollback(TransactionClientChannel.java:91)
      	... 58 more
      
      2019-04-02 21:22:41,709 WARN  [com.arjuna.ats.arjuna] (default task-2) ARJUNA012089: Top-level abort of action 0:ffff7f000001:-70ba203e:5ca3d272:2f received heuristic decision: TwoPhaseOutcome.HEURISTIC_HAZARD
      

      TLDR;
      Server2 is halted and Server1 gets XAER_RMERR. During abort/rollback Server2 is still down and Server1 gets XAER_RMERR. Result of this is that the transaction is in heuristic state and not rolled back.

      Based of the XA spec (http://pubs.opengroup.org/onlinepubs/009680699/toc.pdf) the XAER_RMFAIL (or XA_RETRY) is the correct XAException error code.

      XAER_RMERR
      a resource manager error occurred in the transaction branch

      XAER_RMFAIL
      resource manager unavailable

      XA_RETRY
      routine returned with no effect and may be reissued

              ochaloup@redhat.com Ondrej Chaloupka (Inactive)
              istraka@redhat.com Ivan Straka
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: