Uploaded image for project: 'WildFly'
  1. WildFly
  2. WFLY-18214

Deadlock can happen during JMS authentication with custom realm which requests the same server

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • 30.0.0.Final
    • None
    • JMS, Security
    • None

      There is a possibility of a deadlock during JMS authentication under specific conditions:

      • messaging subsystem is secured with custom security realm
      • the custom security realm is sending requests to the authorization service
      • the authorization service is deployed on the same WildFly instance as the messaging subsystem is running

      Sample scenario where the deadlock occurs:

      1. the jms client publishes a message on a topic
      2. messaging subsystem authenticates the publish using custom security realm
      3. the security realm sends a request to the authorization service

      When the deadlock occurs following exception is thrown:

      Caused by: javax.jms.JMSRuntimeException: AMQ219014: Timed out after waiting 30,000 ms for response when sending packet -18
      	at org.apache.activemq.artemis.jms.client.JmsExceptionUtils.convertToRuntimeException(JmsExceptionUtils.java:88)
      	at org.apache.activemq.artemis.jms.client.ActiveMQConnectionFactory.createContext(ActiveMQConnectionFactory.java:326)
      	at org.apache.activemq.artemis.jms.client.ActiveMQConnectionFactory.createContext(ActiveMQConnectionFactory.java:314)
      	at com.sample.test.PublishIT.lambda$testSequentialPublish$0(PublishIT.java:36)
      	at org.junit.jupiter.api.AssertDoesNotThrow.assertDoesNotThrow(AssertDoesNotThrow.java:49)
      	... 74 more
      Caused by: javax.jms.JMSException: AMQ219014: Timed out after waiting 30,000 ms for response when sending packet -18
      	at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:539)
      	at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:443)
      	at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager.createSessionContext(ActiveMQClientProtocolManager.java:306)
      	at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager.createSessionContext(ActiveMQClientProtocolManager.java:254)
      	at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.createSessionChannel(ClientSessionFactoryImpl.java:1437)
      	at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.createSessionInternal(ClientSessionFactoryImpl.java:743)
      	at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.createSession(ClientSessionFactoryImpl.java:324)
      	at org.apache.activemq.artemis.jms.client.ActiveMQConnection.authorize(ActiveMQConnection.java:661)
      	at org.apache.activemq.artemis.jms.client.ActiveMQConnectionFactory.createConnectionInternal(ActiveMQConnectionFactory.java:916)
      	at org.apache.activemq.artemis.jms.client.ActiveMQConnectionFactory.createContext(ActiveMQConnectionFactory.java:321)
      	... 77 more
      Caused by: ActiveMQConnectionTimedOutException[errorType=CONNECTION_TIMEDOUT message=AMQ219014: Timed out after waiting 30,000 ms for response when sending packet -18]
      	... 87 more
      

      Some thoughts after the problem analysis:
      The messaging subsystem authentication is performed on the IO thread - this results in security realm sending its request on the IO thread blocking it. The IO thread that handles the security realm request may receive the task to handle the request itself and this causes deadlock. Please see the stacktrace excerpt from the IO thread:

      "default I/O-3@21068" prio=5 tid=0x7c nid=NA runnable
        java.lang.Thread.State: RUNNABLE
           blocks Thread-1 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@6c3699d0)@21100
            at sun.nio.ch.SocketDispatcher.read0(SocketDispatcher.java:-1)
            at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:46)
            at sun.nio.ch.NioSocketImpl.tryRead(NioSocketImpl.java:261)
            at sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:312)
            at sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:350)
            at sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:803)
            at java.net.Socket$SocketInputStream.read(Socket.java:966)
            (collapsed org.apache.http stack)
            at org.jboss.resteasy.client.jaxrs.engines.ManualClosingApacheHttpClient43Engine.invoke(ManualClosingApacheHttpClient43Engine.java:302)
            at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.invoke(ClientInvocation.java:494)
            at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.invoke(ClientInvocation.java:69)
            at org.jboss.resteasy.client.jaxrs.internal.ClientInvocationBuilder.get(ClientInvocationBuilder.java:190)
            at com.sample.security.realm.CustomSecurityRealm.getRealmIdentity(CustomSecurityRealm.java:67)
            at org.wildfly.security.auth.server.ServerAuthenticationContext.assignName(ServerAuthenticationContext.java:1223)
            at org.wildfly.security.auth.server.ServerAuthenticationContext$UnassignedState.setPrincipal(ServerAuthenticationContext.java:1767)
            at org.wildfly.security.auth.server.ServerAuthenticationContext$InactiveState.setPrincipal(ServerAuthenticationContext.java:1456)
            at org.wildfly.security.auth.server.ServerAuthenticationContext.setAuthenticationPrincipal(ServerAuthenticationContext.java:416)
            at org.wildfly.security.auth.server.ServerAuthenticationContext.setAuthenticationName(ServerAuthenticationContext.java:390)
            at org.wildfly.security.auth.server.ServerAuthenticationContext.setAuthenticationName(ServerAuthenticationContext.java:374)
            at org.wildfly.extension.messaging.activemq.ElytronSecurityManager.authenticate(ElytronSecurityManager.java:114)
            at org.wildfly.extension.messaging.activemq.ElytronSecurityManager.validateUser(ElytronSecurityManager.java:62)
            at org.apache.activemq.artemis.core.security.impl.SecurityStoreImpl.authenticate(SecurityStoreImpl.java:184)
            at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.createSession(ActiveMQServerImpl.java:1681)
            at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQPacketHandler.handleCreateSession(ActiveMQPacketHandler.java:183)
            at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQPacketHandler.handlePacket(ActiveMQPacketHandler.java:97)
            at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:820)
            at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:426)
            - locked <0x529e> (a java.lang.Object)
            at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:394)
            at org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:682)
            at org.apache.activemq.artemis.core.remoting.impl.netty.ActiveMQChannelHandler.channelRead(ActiveMQChannelHandler.java:73)
            (collapsed io.netty stack)
            at org.xnio.netty.transport.AbstractXnioSocketChannel$ReadListener.handleEvent(AbstractXnioSocketChannel.java:442)
            at org.xnio.netty.transport.AbstractXnioSocketChannel$ReadListener.handleEvent(AbstractXnioSocketChannel.java:378)
            at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
            at org.xnio.conduits.ReadReadyHandler$ChannelListenerHandler.readReady(ReadReadyHandler.java:66)
            at org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:89)
            at org.xnio.nio.WorkerThread.run(WorkerThread.java:591)
      

      The default I/O-3 thread waits for the data from the server and the memory dump looks like this:

      toString(w.name) | (w.selectorWorkQueue.tail - w.selectorWorkQueue.head)
      -------------------------------------------------------------------------
      XNIO-1 Accept    |                                                     0
      XNIO-1 I/O-1     |                                                     0
      default Accept   |                                                     0
      default I/O-1    |                                                     0
      default I/O-2    |                                                     0
      default I/O-3    |                                                     2
      default I/O-4    |                                                     0
      default I/O-5    |                                                     0
      default I/O-6    |                                                     0
      default I/O-7    |                                                     0
      management Accept|                                                     0
      management I/O-1 |                                                     0
      management I/O-2 |                                                     0
      -------------------------------------------------------------------------
      

      There is only one thread that has more than 0 elements in the selectorWorkQueue and it is the same thread as the one handling the http request. The selectorWorkQueue from I/O-3 consists of:

      • QueuedNioTcpServer.acceptTask lambda
      • WorkerThread$SynchTask

            ehugonne1@redhat.com Emmanuel Hugonnet
            psi_bpiotrowski Bartosz Piotrowski (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: