Uploaded image for project: 'JBoss Enterprise Application Platform'
  1. JBoss Enterprise Application Platform
  2. JBEAP-11449

Legacy EJB client: Cannot proceed with invocation since the locator has an affinity on node X which has been excluded from current invocation context

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Verified (View Workflow)
    • Priority: Blocker
    • Resolution: Done
    • Affects Version/s: 7.1.0.DR19
    • Fix Version/s: 7.1.0.CR3
    • Component/s: Clustering, EJB
    • Labels:
      None
    • Target Release:
    • Steps to Reproduce:
      Hide

      1. git clone https://github.com/mvinkler/mock-artifacts-clustering.git
      cd mock-artifacts-clustering/ejbclient/eap7.1-legacy-client/

      2. unzip 2 EAP distributions and run it with standalone-ha.xml profile:

        ${EAP71_HOME}/bin/standalone.sh -Djboss.node.name=node1 -c standalone-ha.xml
        ${EAP71_HOME}/bin/standalone.sh -Djboss.node.name=node2 -c standalone-ha.xml -Djboss.socket.binding.port-offset=100
      

      3. build server project, deploy it (manually) to both servers

      4. build and run client project using

        mvn clean package -Pejbclient3x
        mvn exec:exec -Pejbclient3x
      

      5. press ENTER few times in the terminal with the running client, invocations go to one of the two nodes (each time the same node)
      6. kill/shutdown the server which handled the invocations
      7. press ENTER one more time - the client should get "Connection refused" and retry on the second node, it does not

      Show
      1. git clone https://github.com/mvinkler/mock-artifacts-clustering.git cd mock-artifacts-clustering/ejbclient/eap7.1-legacy-client/ 2. unzip 2 EAP distributions and run it with standalone-ha.xml profile: ${EAP71_HOME}/bin/standalone.sh -Djboss.node.name=node1 -c standalone-ha.xml ${EAP71_HOME}/bin/standalone.sh -Djboss.node.name=node2 -c standalone-ha.xml -Djboss.socket.binding.port-offset=100 3. build server project, deploy it (manually) to both servers 4. build and run client project using mvn clean package -Pejbclient3x mvn exec:exec -Pejbclient3x 5. press ENTER few times in the terminal with the running client, invocations go to one of the two nodes (each time the same node) 6. kill/shutdown the server which handled the invocations 7. press ENTER one more time - the client should get "Connection refused" and retry on the second node, it does not

      Description

      We saw this error on client during failover scenarios where the legacy EJB client was used. All the ejbremote and ejbservlet scenarios (shutdown, jvmkill and undeploy) with legacy EJB client are affected.

      After further investigation, it is now clear this issue results in failover not working when using legacy (3.x) EJB client library with EAP 7.1.

      Client log stacktrace:

      2017/06/08 07:58:13:998 EDT [ERROR][Runner - 1585] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Error getting response. <java.lang.IllegalStateException: EJBCLIENT000051: Cannot proceed with invocation since the locator StatefulEJBLocator for "clusterbench-ee7/clusterbench-ee7-ejb/RemoteStatefulSBImpl", view is interface org.jboss.test.clusterbench.ejb.stateful.RemoteStatefulSB, affinity is Node "perf18", session ID is UnknownSessionID [48576648514969545370506965655252665656546767705252535567656766545365] has an affinity on node perf18 which has been excluded from current invocation context org.jboss.ejb.client.EJBClientInvocationContext@7ad3d631>
              java.lang.IllegalStateException: EJBCLIENT000051: Cannot proceed with invocation since the locator StatefulEJBLocator for "clusterbench-ee7/clusterbench-ee7-ejb/RemoteStatefulSBImpl", view is interface org.jboss.test.clusterbench.ejb.stateful.RemoteStatefulSB, affinity is Node "perf18", session ID is UnknownSessionID [48576648514969545370506965655252665656546767705252535567656766545365] has an affinity on node perf18 which has been excluded from current invocation context org.jboss.ejb.client.EJBClientInvocationContext@7ad3d631
      	at org.jboss.ejb.client.ReceiverInterceptor.handleInvocation(ReceiverInterceptor.java:66)
      	at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:186)
      	at org.jboss.ejb.client.EJBClientInvocationContext.retryRequest(EJBClientInvocationContext.java:211)
      	at org.jboss.ejb.client.EJBReceiverInvocationContext.retryInvocation(EJBReceiverInvocationContext.java:95)
      	at org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler.processMessage(NoSuchEJBExceptionResponseHandler.java:78)
      	at org.jboss.ejb.client.remoting.ChannelAssociation.processResponse(ChannelAssociation.java:395)
      	at org.jboss.ejb.client.remoting.ChannelAssociation$ResponseReceiver.handleMessage(ChannelAssociation.java:507)
      	at org.jboss.remoting3.remote.RemoteConnectionChannel.lambda$handleMessageData$3(RemoteConnectionChannel.java:434)
      	at org.jboss.remoting3.EndpointImpl$TrackingExecutor.lambda$execute$0(EndpointImpl.java:902)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      	at ...asynchronous invocation...(Unknown Source)
      	at org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler$ResultProducer.getResult(NoSuchEJBExceptionResponseHandler.java:105)
      	at org.jboss.ejb.client.EJBClientInvocationContext$1.run(EJBClientInvocationContext.java:282)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:279)
      	at org.jboss.ejb.client.EJBObjectInterceptor.handleInvocationResult(EJBObjectInterceptor.java:64)
      	at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:290)
      	at org.jboss.ejb.client.EJBHomeInterceptor.handleInvocationResult(EJBHomeInterceptor.java:88)
      	at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:290)
      	at org.jboss.ejb.client.TransactionInterceptor.handleInvocationResult(TransactionInterceptor.java:46)
      	at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:290)
      	at org.jboss.ejb.client.ReceiverInterceptor.handleInvocationResult(ReceiverInterceptor.java:142)
      	at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:265)
      	at org.jboss.ejb.client.EJBClientInvocationContext.awaitResponse(EJBClientInvocationContext.java:453)
      	at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:204)
      	at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:183)
      	at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:146)
      	at com.sun.proxy.$Proxy2.getSerialAndIncrement(Unknown Source)
      	at org.jboss.smartfrog.clustering.ejb3.StatefulSBProcessorFactoryImplLegacyClient$EJB3RequestProcessor.processRequest(StatefulSBProcessorFactoryImplLegacyClient.java:79)
      	at org.jboss.smartfrog.loaddriver.CompoundRequestProcessorFactoryImpl$CompoundRequestProcessor.processRequest(CompoundRequestProcessorFactoryImpl.java:52)
      	at org.jboss.smartfrog.loaddriver.Runner.run(Runner.java:103)
      	at java.lang.Thread.run(Thread.java:745)
      

      The exception was preceded and followed by INFO message:

      07:58:13,998 INFO  [org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler] (Remoting "config-based-ejb-client-endpoint" task-14) Retrying invocation which failed on node perf18 with exception:
      javax.ejb.NoSuchEJBException: No such EJB: clusterbench-ee7/clusterbench-ee7-ejb/RemoteStatefulSBImpl
      	at org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler.processMessage(NoSuchEJBExceptionResponseHandler.java:64)
      	at org.jboss.ejb.client.remoting.ChannelAssociation.processResponse(ChannelAssociation.java:395)
      	at org.jboss.ejb.client.remoting.ChannelAssociation$ResponseReceiver.handleMessage(ChannelAssociation.java:507)
      	at org.jboss.remoting3.remote.RemoteConnectionChannel.lambda$receiveMessage$2(RemoteConnectionChannel.java:365)
      	at org.jboss.remoting3.EndpointImpl$TrackingExecutor.lambda$execute$0(EndpointImpl.java:902)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      

      This error was logged during the node18 shutdown:

      2017/06/08 07:58:13:985 EDT [DEBUG][RMI TCP Connection(53)-10.16.90.52] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - [SHUTDOWN] JBossShutdown command executed successfully.
      2017/06/08 07:58:13:986 EDT [DEBUG][RMI TCP Connection(53)-10.16.90.52] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Waiting for server to shutdown.
      

      Link to client log:
      http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_eap-7x-failover-legacy-ejb-ejbremote-shutdown-dist-async/3/console-perf17/

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              swd847 Stuart Douglas
              Reporter:
              dcihak Daniel Cihak
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: