Uploaded image for project: 'JBoss Transaction Manager'
  1. JBoss Transaction Manager
  2. JBTM-3006

Tooling delete operation can left hanging remote txn branch when it was subordinate prepared participant

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Won't Do
    • Icon: Minor Minor
    • None
    • 5.8.0.Final
    • Tooling
    • None
    • Hide

      It's possible to reproduce with attached data-subordnate.zip and WFLY started with the attached standalone-full.xml.

      • download and unzip WFLY12
      • unzip the zip and move it under $WFLY_HOME/standalone directory (the unzipped directory is named data)
      • copy standalone-full-jms.xml to $WFLY_HOME/standalone/configuration
      • run $WFLY_HOME/bin/standalone.sh -c standalone-full-jms.xml -Dcom.arjuna.ats.arjuna.recovery.periodicRecoveryPeriod=9999
      • in other console run $WFLY_HOME/bin/jboss-cli.sh -c and probe the transaction log store /subsystem=transactions/log-store=log-store:probe
      • delete the existing transaction with command like /subsystem=transactions/log-store=log-store/transactions=0\:ffff0a2805b6\:-19805d73\:5ab0d062\:1c:delete()
      • check the server.log and look for the XAException
      Show
      It's possible to reproduce with attached data-subordnate.zip and WFLY started with the attached standalone-full.xml . download and unzip WFLY12 unzip the zip and move it under $WFLY_HOME/standalone directory (the unzipped directory is named data ) copy standalone-full-jms.xml to $WFLY_HOME/standalone/configuration run $WFLY_HOME/bin/standalone.sh -c standalone-full-jms.xml -Dcom.arjuna.ats.arjuna.recovery.periodicRecoveryPeriod=9999 in other console run $WFLY_HOME/bin/jboss-cli.sh -c and probe the transaction log store /subsystem=transactions/log-store=log-store:probe delete the existing transaction with command like /subsystem=transactions/log-store=log-store/transactions=0\:ffff0a2805b6\:-19805d73\:5ab0d062\:1c:delete() check the server.log and look for the XAException

      When administrator decides to call 'delete' in WFLY (or as jmx mbean call) and the transaction is subordinate (meaning started by another server and the context was propagated to this one) then if they are in heuristic state the transaction is removed from the narayana object store and from the RA. If the transaction is in prepared state then participant is removed from narayana object store (that could be influenced by settings of ignoreMBeanHeuristics property, see https://github.com/jbosstm/narayana/blob/master/ArjunaCore/arjuna/classes/com/arjuna/ats/arjuna/common/ObjectStoreEnvironmentBean.java#L804) but the transaction branch at the RA is left untouched - XAResource.forget is called but it fails at the participant remote branch (see [XA spec|pubs.opengroup.org/onlinepubs/009680699/toc.pdf) - the XA spec permits calling forget only on heuristically finished branches but this is still prepared.

      The final outcome is that there is removed record from Narayana object store and left untouched in-doubt (prepared) remote branch.

      Such record should be handled by recovery (ideally) the SubordinateJTAXAResourceOrphanFilter votes for rollback but then the SubordinationManagerXAResourceOrphanFilter forces to leave it as it is. Thus such record will be handing forever.

       TRACE [com.arjuna.ats.arjuna] (Periodic Recovery) OutputObjectState::OutputObjectState()
 DEBUG [com.arjuna.ats.jta] (Periodic Recovery) No record found for < formatId=131077, gtrid_length=29, bqual_length=37, tx_uid=0:ffff0a280466:-51bd8cd:5aa92b2b:1c, node_name=1, branch_uid=0:ffff0a280466:758014d3:5aa92b2f:26, subordinatenodename=2, eis_name=java:/JmsXA NodeId:2bda713c-2790-11e8-8b4f-0a0027000000 >
 DEBUG [com.arjuna.ats.jta] (Periodic Recovery) XAResourceOrphanFilter com.arjuna.ats.internal.jta.recovery.arjunacore.SubordinateJTAXAResourceOrphanFilter voted ROLLBACK
 DEBUG [com.arjuna.ats.jta] (Periodic Recovery) subordinate node name of < formatId=131077, gtrid_length=29, bqual_length=37, tx_uid=0:ffff0a280466:-51bd8cd:5aa92b2b:1c, node_name=1, branch_uid=0:ffff0a280466:758014d3:5aa92b2f:26, subordinatenodename=2, eis_name=java:/JmsXA NodeId:2bda713c-2790-11e8-8b4f-0a0027000000 > is 2
 TRACE [com.arjuna.ats.jta] (Periodic Recovery) TransactionImple.getStatus: javax.transaction.Status.STATUS_UNKNOWN
 DEBUG [com.arjuna.ats.jta] (Periodic Recovery) XAResourceOrphanFilter com.arjuna.ats.internal.jta.recovery.arjunacore.SubordinationManagerXAResourceOrphanFilter voted LEAVE_ALONE
 TRACE [com.arjuna.ats.arjuna] (Periodic Recovery) RecoveryXids isStale Check org.jboss.activemq.artemis.wildfly.integration.WildFlyActiveMQXAResourceWrapper@3397522a 1521650139598 1521650140600 true

        2. jboss-cli-commands.txt
          2 kB
          Ondrej Chaloupka
        3. standalone-full.xml
          31 kB
          Ondrej Chaloupka

              ochaloup@redhat.com Ondrej Chaloupka (Inactive)
              ochaloup@redhat.com Ondrej Chaloupka (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated:
                Resolved: