Loading...

Type: Bug
Resolution: Done
Priority: Blocker
Fix Version/s: 7.1.0.CR3
Affects Version/s: 7.1.0.DR19
Component/s: Clustering, EJB
Labels:
None

CDW devel_ack:
CDW docs_ack:
CDW pm_ack:
CDW qa_ack:
CDW release:
Target Release:

7.1.0.GA
Steps to Reproduce:
Hide

1. git clone https://github.com/mvinkler/mock-artifacts-clustering.git
cd mock-artifacts-clustering/ejbclient/eap7.1-legacy-client/

2. unzip 2 EAP distributions and run it with standalone-ha.xml profile:

${EAP71_HOME}/bin/standalone.sh -Djboss.node.name=node1 -c standalone-ha.xml ${EAP71_HOME}/bin/standalone.sh -Djboss.node.name=node2 -c standalone-ha.xml -Djboss.socket.binding.port-offset=100

3. build server project, deploy it (manually) to both servers

4. build and run client project using

mvn clean package -Pejbclient3x mvn exec:exec -Pejbclient3x

5. press ENTER few times in the terminal with the running client, invocations go to one of the two nodes (each time the same node)
6. kill/shutdown the server which handled the invocations
7. press ENTER one more time - the client should get "Connection refused" and retry on the second node, it does not
Show
1. git clone https://github.com/mvinkler/mock-artifacts-clustering.git cd mock-artifacts-clustering/ejbclient/eap7.1-legacy-client/ 2. unzip 2 EAP distributions and run it with standalone-ha.xml profile: ${EAP71_HOME}/bin/standalone.sh -Djboss.node.name=node1 -c standalone-ha.xml ${EAP71_HOME}/bin/standalone.sh -Djboss.node.name=node2 -c standalone-ha.xml -Djboss.socket.binding.port-offset=100 3. build server project, deploy it (manually) to both servers 4. build and run client project using mvn clean package -Pejbclient3x mvn exec:exec -Pejbclient3x 5. press ENTER few times in the terminal with the running client, invocations go to one of the two nodes (each time the same node) 6. kill/shutdown the server which handled the invocations 7. press ENTER one more time - the client should get "Connection refused" and retry on the second node, it does not

SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

We saw this error on client during failover scenarios where the legacy EJB client was used. All the ejbremote and ejbservlet scenarios (shutdown, jvmkill and undeploy) with legacy EJB client are affected.

After further investigation, it is now clear this issue results in failover not working when using legacy (3.x) EJB client library with EAP 7.1.

Client log stacktrace:

2017/06/08 07:58:13:998 EDT [ERROR][Runner - 1585] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Error getting response. <java.lang.IllegalStateException: EJBCLIENT000051: Cannot proceed with invocation since the locator StatefulEJBLocator for "clusterbench-ee7/clusterbench-ee7-ejb/RemoteStatefulSBImpl", view is interface org.jboss.test.clusterbench.ejb.stateful.RemoteStatefulSB, affinity is Node "perf18", session ID is UnknownSessionID [48576648514969545370506965655252665656546767705252535567656766545365] has an affinity on node perf18 which has been excluded from current invocation context org.jboss.ejb.client.EJBClientInvocationContext@7ad3d631>
        java.lang.IllegalStateException: EJBCLIENT000051: Cannot proceed with invocation since the locator StatefulEJBLocator for "clusterbench-ee7/clusterbench-ee7-ejb/RemoteStatefulSBImpl", view is interface org.jboss.test.clusterbench.ejb.stateful.RemoteStatefulSB, affinity is Node "perf18", session ID is UnknownSessionID [48576648514969545370506965655252665656546767705252535567656766545365] has an affinity on node perf18 which has been excluded from current invocation context org.jboss.ejb.client.EJBClientInvocationContext@7ad3d631
	at org.jboss.ejb.client.ReceiverInterceptor.handleInvocation(ReceiverInterceptor.java:66)
	at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:186)
	at org.jboss.ejb.client.EJBClientInvocationContext.retryRequest(EJBClientInvocationContext.java:211)
	at org.jboss.ejb.client.EJBReceiverInvocationContext.retryInvocation(EJBReceiverInvocationContext.java:95)
	at org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler.processMessage(NoSuchEJBExceptionResponseHandler.java:78)
	at org.jboss.ejb.client.remoting.ChannelAssociation.processResponse(ChannelAssociation.java:395)
	at org.jboss.ejb.client.remoting.ChannelAssociation$ResponseReceiver.handleMessage(ChannelAssociation.java:507)
	at org.jboss.remoting3.remote.RemoteConnectionChannel.lambda$handleMessageData$3(RemoteConnectionChannel.java:434)
	at org.jboss.remoting3.EndpointImpl$TrackingExecutor.lambda$execute$0(EndpointImpl.java:902)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
	at ...asynchronous invocation...(Unknown Source)
	at org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler$ResultProducer.getResult(NoSuchEJBExceptionResponseHandler.java:105)
	at org.jboss.ejb.client.EJBClientInvocationContext$1.run(EJBClientInvocationContext.java:282)
	at java.security.AccessController.doPrivileged(Native Method)
	at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:279)
	at org.jboss.ejb.client.EJBObjectInterceptor.handleInvocationResult(EJBObjectInterceptor.java:64)
	at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:290)
	at org.jboss.ejb.client.EJBHomeInterceptor.handleInvocationResult(EJBHomeInterceptor.java:88)
	at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:290)
	at org.jboss.ejb.client.TransactionInterceptor.handleInvocationResult(TransactionInterceptor.java:46)
	at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:290)
	at org.jboss.ejb.client.ReceiverInterceptor.handleInvocationResult(ReceiverInterceptor.java:142)
	at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:265)
	at org.jboss.ejb.client.EJBClientInvocationContext.awaitResponse(EJBClientInvocationContext.java:453)
	at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:204)
	at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:183)
	at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:146)
	at com.sun.proxy.$Proxy2.getSerialAndIncrement(Unknown Source)
	at org.jboss.smartfrog.clustering.ejb3.StatefulSBProcessorFactoryImplLegacyClient$EJB3RequestProcessor.processRequest(StatefulSBProcessorFactoryImplLegacyClient.java:79)
	at org.jboss.smartfrog.loaddriver.CompoundRequestProcessorFactoryImpl$CompoundRequestProcessor.processRequest(CompoundRequestProcessorFactoryImpl.java:52)
	at org.jboss.smartfrog.loaddriver.Runner.run(Runner.java:103)
	at java.lang.Thread.run(Thread.java:745)

The exception was preceded and followed by INFO message:

07:58:13,998 INFO  [org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler] (Remoting "config-based-ejb-client-endpoint" task-14) Retrying invocation which failed on node perf18 with exception:
javax.ejb.NoSuchEJBException: No such EJB: clusterbench-ee7/clusterbench-ee7-ejb/RemoteStatefulSBImpl
	at org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler.processMessage(NoSuchEJBExceptionResponseHandler.java:64)
	at org.jboss.ejb.client.remoting.ChannelAssociation.processResponse(ChannelAssociation.java:395)
	at org.jboss.ejb.client.remoting.ChannelAssociation$ResponseReceiver.handleMessage(ChannelAssociation.java:507)
	at org.jboss.remoting3.remote.RemoteConnectionChannel.lambda$receiveMessage$2(RemoteConnectionChannel.java:365)
	at org.jboss.remoting3.EndpointImpl$TrackingExecutor.lambda$execute$0(EndpointImpl.java:902)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

This error was logged during the node18 shutdown:

2017/06/08 07:58:13:985 EDT [DEBUG][RMI TCP Connection(53)-10.16.90.52] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - [SHUTDOWN] JBossShutdown command executed successfully.
2017/06/08 07:58:13:986 EDT [DEBUG][RMI TCP Connection(53)-10.16.90.52] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Waiting for server to shutdown.

Link to client log:
http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_eap-7x-failover-legacy-ejb-ejbremote-shutdown-dist-async/3/console-perf17/

is cloned by

EJBCLIENT-286 Legacy client is sending a NodeAffinity instead of a ClusterAffinity for SFSB session creation

Resolved

is incorporated by

JBEAP-13509 Upgrade ejb-client to 4.0.7.Final

Closed

is related to

JBEAP-13490 Legacy EJB client application is unable to authenticate against secured EJB deployment running on server on standalone-ha profile

Closed

Details

Description

Attachments

Issue Links

Easy Agile Planning Poker

Activity

People

Dates