Uploaded image for project: 'JBoss Enterprise Application Platform'
  1. JBoss Enterprise Application Platform
  2. JBEAP-30320

Pod stuck in the SCALING_DOWN_RECOVERY_PROCESSING status during scale-down with HEURISTIC transactions

XMLWordPrintable

    • False
    • Hide

      None

      Show
      None
    • False

      It seems there is inconsistency in statuses between the operator and server during pod scale-down when there are HEURISTIC transactions. The pod is stuck in SCALING_DOWN_RECOVERY_PROCESSING and user never gets information the pod status is actually SCALING_DOWN_RECOVERY_HEURISTICS which need manual intervention.

      While we wait until the pod gets into SCALING_DOWN_RECOVERY_HEURISTICS state there are warnings in the client pod log:

      ARJUNA012035: Activate of object with id = 0:0:0:0:0 and type /StateManager/AbstractRecord/XAResourceRecord unexpectedly failed

       

      We see this exception in the client pod log:

      Caused by: jakarta.transaction.HeuristicMixedException
    at org.jboss.jts//com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1301)
    at org.jboss.jts//com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:128)
    at org.jboss.jts.integration@6.0.6.Final-redhat-00001//com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:95)
    at org.wildfly.transaction.client@3.0.5.Final-redhat-00001//org.wildfly.transaction.client.LocalTransaction.commitAndDissociate(LocalTransaction.java:78)
    at org.wildfly.transaction.client@3.0.5.Final-redhat-00001//org.wildfly.transaction.client.ContextTransactionManager.commit(ContextTransactionManager.java:71)
    at org.jboss.as.ejb3@8.0.10.GA-redhat-00002//org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:91)
    ... 113 more
    Suppressed: javax.transaction.xa.XAException
        at deployment.tx-client.war//org.jboss.as.quickstarts.xa.resources.MockXAResource.commit(MockXAResource.java:129)
        at org.jboss.jts//com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelCommit(XAResourceRecord.java:473)
        at org.jboss.jts//com.arjuna.ats.arjuna.coordinator.BasicAction.doCommit(BasicAction.java:2906)
        at org.jboss.jts//com.arjuna.ats.arjuna.coordinator.BasicAction.doCommit(BasicAction.java:2822)
        at org.jboss.jts//com.arjuna.ats.arjuna.coordinator.BasicAction.phase2Commit(BasicAction.java:1878)
        at org.jboss.jts//com.arjuna.ats.arjuna.coordinator.BasicAction.End(BasicAction.java:1534)
        at org.jboss.jts//com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:96)
        at org.jboss.jts//com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
        at org.jboss.jts//com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1295)

      It is probably caused by https://issues.redhat.com/browse/JBEAP-28831.
      For 8.1 it is fixed by upgrading the operator to an unreleased version. See https://issues.redhat.com/browse/EAPQE-3232 and MR https://gitlab.cee.redhat.com/jbossqe-eap/openshift-eap-tests/-/merge_requests/1291.

      Here we can see failing job https://cp-80x-jenkins.eapqe.psi.redhat.com/job/eap-80x-openshift-4-operator-80-openjdk17/147/.

      Logs com.redhat.xpaas.eap.xa.EjbTxnRemotingScaleDownTest.txt, tx-server-0.log, tx-client-0.log, eap-operator-mdl82.log and eap-operator-58968fc666-tsrmx.log.

        1. com.redhat.xpaas.eap.xa.EjbTxnRemotingScaleDownTest.txt
          7 kB
        2. eap-operator-58968fc666-tsrmx.log
          76 kB
        3. eap-operator-mdl82.log
          0.5 kB
        4. tx-client-0.log
          550 kB
        5. tx-server-0.log
          212 kB

              yborgess1@redhat.com Yeray Borges Santana
              kathermanova Katarína Hermanová
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

                Created:
                Updated:
                Resolved: