Uploaded image for project: 'EJB Client Library (AS7+)'
  1. EJB Client Library (AS7+)
  2. EJBCLIENT-176

EJB client retry mechanisms causing inconsistencies with interceptor chain index when client side and server side retries invoked.

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • 2.1.9.Final
    • 2.1.6.Final
    • None

      In a failover test for Wildfly involving two clusters (one cluster receives EJB client invocations from a client, and forwards those EJB client invocations to a second cluster), we are seeing two types of exceptions relating to inconsistent interceptor chain indexes on the forwarding server client.

      One involves exceptions on sending the invocation:

      2016-11-01 14:09:02,988 WARN  [com.arjuna.ats.arjuna] (default task-100) ARJUNA012091: Top-level abort of action 0:ffff0a108d68:650111a8:5818d989:e871b received TwoPhaseOutcome.FINISH_ERROR from com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord
      2016-11-01 14:09:02,991 ERROR [org.jboss.as.ejb3.invocation] (default task-100) WFLYEJB0034: EJB Invocation failed on component ForwardingBeanImpl for method public abstract java.lang.String org.jboss.qa.eap.clustering.lodh.testApps.june2013.Bean.test(java.lang.String,long): javax.ejb.EJBException: java.lang.IllegalStateException: EJBCLIENT000032: Cannot retry a request which hasn't previously been completed
      	at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInOurTx(CMTTxInterceptor.java:187)
      	at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:281)
      	at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:331)
      	at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:240)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:359)
      	at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:456)
      	at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73)
      	at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:83)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:359)
      	at org.jboss.as.ejb3.remote.EJBRemoteTransactionPropagatingInterceptor.processInvocation(EJBRemoteTransactionPropagatingInterceptor.java:80)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:359)
      	at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:359)
      	at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:359)
      	at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:359)
      	at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:359)
      	at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:359)
      	at org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:53)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:359)
      	at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:359)
      	at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:359)
      	at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:359)
      	at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:359)
      	at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:375)
      	at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:609)
      	at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:359)
      	at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:375)
      	at org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:359)
      	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
      	at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
      	at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.invokeMethod(MethodInvocationMessageHandler.java:333)
      	at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.access$100(MethodInvocationMessageHandler.java:68)
      	at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler$1.run(MethodInvocationMessageHandler.java:202)
      	at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.processMessage(MethodInvocationMessageHandler.java:268)
      	at org.jboss.as.ejb3.remote.protocol.versionone.VersionOneProtocolChannelReceiver.processMessage(VersionOneProtocolChannelReceiver.java:213)
      	at org.jboss.as.ejb3.remote.protocol.versiontwo.VersionTwoProtocolChannelReceiver.processMessage(VersionTwoProtocolChannelReceiver.java:76)
      	at org.jboss.as.ejb3.remote.protocol.versionone.VersionOneProtocolChannelReceiver.handleMessage(VersionOneProtocolChannelReceiver.java:159)
      	at org.jboss.remoting3.remote.RemoteConnectionChannel$5.run(RemoteConnectionChannel.java:456)
      	at org.jboss.remoting3.EndpointImpl$TrackingExecutor$1.run(EndpointImpl.java:731)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: java.lang.IllegalStateException: EJBCLIENT000032: Cannot retry a request which hasn't previously been completed
      	at org.jboss.ejb.client.EJBClientInvocationContext.retryRequest(EJBClientInvocationContext.java:205)
      	at org.jboss.ejb.client.EJBInvocationHandler.sendRequestWithPossibleRetries(EJBInvocationHandler.java:258)
      	at org.jboss.ejb.client.EJBInvocationHandler.sendRequestWithPossibleRetries(EJBInvocationHandler.java:278)
      	at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:200)
      	at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:183)
      	at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:146)
      	at com.sun.proxy.$Proxy71.test(Unknown Source)
      	at org.jboss.qa.eap.clustering.lodh.testApps.june2013.ForwardingBeanImpl.test(ForwardingBeanImpl.java:30)
      	at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:359)
      	at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:456)
      	at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:82)
      	at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:93)
      	at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:359)
      	at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:359)
      	at org.jboss.as.jpa.interceptor.SBInvocationInterceptor.processInvocation(SBInvocationInterceptor.java:47)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:359)
      	at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:359)
      	at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:359)
      	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
      	at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:359)
      	at org.jboss.as.ejb3.component.pool.PooledInstanceInterceptor.processInvocation(PooledInstanceInterceptor.java:51)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:359)
      	at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:279)
      	... 50 more
      

      The other on getting the result:

      2016-11-01 14:09:02,988 WARN  [com.arjuna.ats.arjuna] (default task-100) ARJUNA012091: Top-level abort of action 0:ffff0a108d68:650111a8:5818d989:e871b received TwoPhaseOutcome.FINISH_ERROR from com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord
      2016-11-01 14:09:02,991 ERROR [org.jboss.as.ejb3.invocation] (default task-100) WFLYEJB0034: EJB Invocation failed on component ForwardingBeanImpl for method public abstract java.lang.String org.jboss.qa.eap.clustering.lodh.testApps.june2013.Bean.test(java.lang.String,long): javax.ejb.EJBException: java.lang.IllegalStateException: EJBCLIENT000032: Cannot retry a request which hasn't previously been completed
      	at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInOurTx(CMTTxInterceptor.java:187)
      	at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:281)
      	at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:331)
      	at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:240)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:359)
      	at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:456)
      	at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73)
      	at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:83)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:359)
      	at org.jboss.as.ejb3.remote.EJBRemoteTransactionPropagatingInterceptor.processInvocation(EJBRemoteTransactionPropagatingInterceptor.java:80)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:359)
      	at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:359)
      	at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:359)
      	at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:359)
      	at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:359)
      	at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:359)
      	at org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:53)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:359)
      	at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:359)
      	at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:359)
      	at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:359)
      	at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:359)
      	at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:375)
      	at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:609)
      	at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:359)
      	at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:375)
      	at org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:359)
      	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
      	at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
      	at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.invokeMethod(MethodInvocationMessageHandler.java:333)
      	at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.access$100(MethodInvocationMessageHandler.java:68)
      	at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler$1.run(MethodInvocationMessageHandler.java:202)
      	at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.processMessage(MethodInvocationMessageHandler.java:268)
      	at org.jboss.as.ejb3.remote.protocol.versionone.VersionOneProtocolChannelReceiver.processMessage(VersionOneProtocolChannelReceiver.java:213)
      	at org.jboss.as.ejb3.remote.protocol.versiontwo.VersionTwoProtocolChannelReceiver.processMessage(VersionTwoProtocolChannelReceiver.java:76)
      	at org.jboss.as.ejb3.remote.protocol.versionone.VersionOneProtocolChannelReceiver.handleMessage(VersionOneProtocolChannelReceiver.java:159)
      	at org.jboss.remoting3.remote.RemoteConnectionChannel$5.run(RemoteConnectionChannel.java:456)
      	at org.jboss.remoting3.EndpointImpl$TrackingExecutor$1.run(EndpointImpl.java:731)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: java.lang.IllegalStateException: EJBCLIENT000032: Cannot retry a request which hasn't previously been completed
      	at org.jboss.ejb.client.EJBClientInvocationContext.retryRequest(EJBClientInvocationContext.java:205)
      	at org.jboss.ejb.client.EJBInvocationHandler.sendRequestWithPossibleRetries(EJBInvocationHandler.java:258)
      	at org.jboss.ejb.client.EJBInvocationHandler.sendRequestWithPossibleRetries(EJBInvocationHandler.java:278)
      	at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:200)
      	at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:183)
      	at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:146)
      	at com.sun.proxy.$Proxy71.test(Unknown Source)
      	at org.jboss.qa.eap.clustering.lodh.testApps.june2013.ForwardingBeanImpl.test(ForwardingBeanImpl.java:30)
      	at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:359)
      	at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:456)
      	at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:82)
      	at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:93)
      	at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:359)
      	at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:359)
      	at org.jboss.as.jpa.interceptor.SBInvocationInterceptor.processInvocation(SBInvocationInterceptor.java:47)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:359)
      	at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:359)
      	at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:359)
      	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
      	at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:359)
      	at org.jboss.as.ejb3.component.pool.PooledInstanceInterceptor.processInvocation(PooledInstanceInterceptor.java:51)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:359)
      	at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:279)
      	... 50 more
      
      

              rachmato@redhat.com Richard Achmatowicz
              rachmato@redhat.com Richard Achmatowicz
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: