Details
-
Bug
-
Resolution: Done
-
Major
-
None
-
None
Description
There is a possibility of a deadlock during JMS authentication under specific conditions:
- messaging subsystem is secured with custom security realm
- the custom security realm is sending requests to the authorization service
- the authorization service is deployed on the same WildFly instance as the messaging subsystem is running
Sample scenario where the deadlock occurs:
- the jms client publishes a message on a topic
- messaging subsystem authenticates the publish using custom security realm
- the security realm sends a request to the authorization service
When the deadlock occurs following exception is thrown:
Caused by: javax.jms.JMSRuntimeException: AMQ219014: Timed out after waiting 30,000 ms for response when sending packet -18 at org.apache.activemq.artemis.jms.client.JmsExceptionUtils.convertToRuntimeException(JmsExceptionUtils.java:88) at org.apache.activemq.artemis.jms.client.ActiveMQConnectionFactory.createContext(ActiveMQConnectionFactory.java:326) at org.apache.activemq.artemis.jms.client.ActiveMQConnectionFactory.createContext(ActiveMQConnectionFactory.java:314) at com.sample.test.PublishIT.lambda$testSequentialPublish$0(PublishIT.java:36) at org.junit.jupiter.api.AssertDoesNotThrow.assertDoesNotThrow(AssertDoesNotThrow.java:49) ... 74 more Caused by: javax.jms.JMSException: AMQ219014: Timed out after waiting 30,000 ms for response when sending packet -18 at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:539) at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:443) at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager.createSessionContext(ActiveMQClientProtocolManager.java:306) at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager.createSessionContext(ActiveMQClientProtocolManager.java:254) at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.createSessionChannel(ClientSessionFactoryImpl.java:1437) at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.createSessionInternal(ClientSessionFactoryImpl.java:743) at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.createSession(ClientSessionFactoryImpl.java:324) at org.apache.activemq.artemis.jms.client.ActiveMQConnection.authorize(ActiveMQConnection.java:661) at org.apache.activemq.artemis.jms.client.ActiveMQConnectionFactory.createConnectionInternal(ActiveMQConnectionFactory.java:916) at org.apache.activemq.artemis.jms.client.ActiveMQConnectionFactory.createContext(ActiveMQConnectionFactory.java:321) ... 77 more Caused by: ActiveMQConnectionTimedOutException[errorType=CONNECTION_TIMEDOUT message=AMQ219014: Timed out after waiting 30,000 ms for response when sending packet -18] ... 87 more
Some thoughts after the problem analysis:
The messaging subsystem authentication is performed on the IO thread - this results in security realm sending its request on the IO thread blocking it. The IO thread that handles the security realm request may receive the task to handle the request itself and this causes deadlock. Please see the stacktrace excerpt from the IO thread:
"default I/O-3@21068" prio=5 tid=0x7c nid=NA runnable java.lang.Thread.State: RUNNABLE blocks Thread-1 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@6c3699d0)@21100 at sun.nio.ch.SocketDispatcher.read0(SocketDispatcher.java:-1) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:46) at sun.nio.ch.NioSocketImpl.tryRead(NioSocketImpl.java:261) at sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:312) at sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:350) at sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:803) at java.net.Socket$SocketInputStream.read(Socket.java:966) (collapsed org.apache.http stack) at org.jboss.resteasy.client.jaxrs.engines.ManualClosingApacheHttpClient43Engine.invoke(ManualClosingApacheHttpClient43Engine.java:302) at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.invoke(ClientInvocation.java:494) at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.invoke(ClientInvocation.java:69) at org.jboss.resteasy.client.jaxrs.internal.ClientInvocationBuilder.get(ClientInvocationBuilder.java:190) at com.sample.security.realm.CustomSecurityRealm.getRealmIdentity(CustomSecurityRealm.java:67) at org.wildfly.security.auth.server.ServerAuthenticationContext.assignName(ServerAuthenticationContext.java:1223) at org.wildfly.security.auth.server.ServerAuthenticationContext$UnassignedState.setPrincipal(ServerAuthenticationContext.java:1767) at org.wildfly.security.auth.server.ServerAuthenticationContext$InactiveState.setPrincipal(ServerAuthenticationContext.java:1456) at org.wildfly.security.auth.server.ServerAuthenticationContext.setAuthenticationPrincipal(ServerAuthenticationContext.java:416) at org.wildfly.security.auth.server.ServerAuthenticationContext.setAuthenticationName(ServerAuthenticationContext.java:390) at org.wildfly.security.auth.server.ServerAuthenticationContext.setAuthenticationName(ServerAuthenticationContext.java:374) at org.wildfly.extension.messaging.activemq.ElytronSecurityManager.authenticate(ElytronSecurityManager.java:114) at org.wildfly.extension.messaging.activemq.ElytronSecurityManager.validateUser(ElytronSecurityManager.java:62) at org.apache.activemq.artemis.core.security.impl.SecurityStoreImpl.authenticate(SecurityStoreImpl.java:184) at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.createSession(ActiveMQServerImpl.java:1681) at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQPacketHandler.handleCreateSession(ActiveMQPacketHandler.java:183) at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQPacketHandler.handlePacket(ActiveMQPacketHandler.java:97) at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:820) at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:426) - locked <0x529e> (a java.lang.Object) at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:394) at org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:682) at org.apache.activemq.artemis.core.remoting.impl.netty.ActiveMQChannelHandler.channelRead(ActiveMQChannelHandler.java:73) (collapsed io.netty stack) at org.xnio.netty.transport.AbstractXnioSocketChannel$ReadListener.handleEvent(AbstractXnioSocketChannel.java:442) at org.xnio.netty.transport.AbstractXnioSocketChannel$ReadListener.handleEvent(AbstractXnioSocketChannel.java:378) at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92) at org.xnio.conduits.ReadReadyHandler$ChannelListenerHandler.readReady(ReadReadyHandler.java:66) at org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:89) at org.xnio.nio.WorkerThread.run(WorkerThread.java:591)
The default I/O-3 thread waits for the data from the server and the memory dump looks like this:
toString(w.name) | (w.selectorWorkQueue.tail - w.selectorWorkQueue.head) ------------------------------------------------------------------------- XNIO-1 Accept | 0 XNIO-1 I/O-1 | 0 default Accept | 0 default I/O-1 | 0 default I/O-2 | 0 default I/O-3 | 2 default I/O-4 | 0 default I/O-5 | 0 default I/O-6 | 0 default I/O-7 | 0 management Accept| 0 management I/O-1 | 0 management I/O-2 | 0 -------------------------------------------------------------------------
There is only one thread that has more than 0 elements in the selectorWorkQueue and it is the same thread as the one handling the http request. The selectorWorkQueue from I/O-3 consists of:
- QueuedNioTcpServer.acceptTask lambda
- WorkerThread$SynchTask
Attachments
Issue Links
- is incorporated by
-
WFLY-17309 Upgrade Apache Artemis to 2.31.0
-
- Closed
-
- links to
