Uploaded image for project: 'Application Server 7'
  1. Application Server 7
  2. AS7-4274

Remoting deadlock after deserialization exception of a large object graph returned from EJB.

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • 7.1.2.Final (EAP)
    • 7.1.1.Final
    • EJB, Remoting
    • None
    • Hide

      Install/Unpack

      1. Unpack ejb-deadlock.jar to a local directory
      2. cd ejb-deadlock

      Deploy

      1. deploy target/lib/ejb-deadlock.jar to AS 7.1.1.Final

      Running the client

      The client se.rl.deadlock.CustomerBeanClient reproduces the issue by using Byteman to throw a fake ClassNotFoundException. The Byteman configuration can be found in src/test/resources/customer-bean-client.btm.

      The client can be run from the command line via Gradle:

      ./gradlew exec

      Or run from within an IDE, but requires setting the following VM argument:

      -javaagent:lib/byteman.jar=script:src/test/resources/customer-bean-client.btm

      The source will be attached but can also be found on github

      standalone/conf/standalone.xml

      Under

      <subsystem xmlns="urn:jboss:domain:remoting:1.1">

      Add

      <outbound-connections>
  <remote-outbound-connection name="remote-ejb-connection1" outbound-socket-binding-ref="remote-ejb">
  </remote-outbound-connection>
  <remote-outbound-connection name="remote-ejb-connection2" outbound-socket-binding-ref="remote-ejb">
  </remote-outbound-connection>
</outbound-connections>

      and

      Under

      <socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">

      Add

      <outbound-socket-binding name="remote-ejb">
  <remote-destination host="localhost" port="4447"/>
</outbound-socket-binding>
      Show
      Install/Unpack Unpack ejb-deadlock.jar to a local directory cd ejb-deadlock Deploy deploy target/lib/ejb-deadlock.jar to AS 7.1.1.Final Running the client The client se.rl.deadlock.CustomerBeanClient reproduces the issue by using Byteman to throw a fake ClassNotFoundException. The Byteman configuration can be found in src/test/resources/customer-bean-client.btm . The client can be run from the command line via Gradle: ./gradlew exec Or run from within an IDE, but requires setting the following VM argument: -javaagent:lib/byteman.jar=script:src/test/resources/customer-bean-client.btm The source will be attached but can also be found on github standalone/conf/standalone.xml Under <subsystem xmlns= "urn:jboss:domain:remoting:1.1" > Add <outbound-connections> <remote-outbound-connection name= "remote-ejb-connection1" outbound-socket-binding-ref= "remote-ejb" > </remote-outbound-connection> <remote-outbound-connection name= "remote-ejb-connection2" outbound-socket-binding-ref= "remote-ejb" > </remote-outbound-connection> </outbound-connections> and Under <socket-binding-group name= "standard-sockets" default - interface = " public " port-offset= "${jboss.socket.binding.port-offset:0}" > Add <outbound-socket-binding name= "remote-ejb" > <remote-destination host= "localhost" port= "4447" /> </outbound-socket-binding>

      The use case is a remote EJB client that calls an EJB which returns a large object graph. Due to a configuration error on our part a class that is part of this object graph is not available in the clients classpath. This causes a ClassNotFoundException to be thrown when deserializing, and after this exception is thrown the client "freezes" but can be interrupted by CTRL+C, but the server will deadlock and will not respond and has to be shutdown by killing the server process.

      Below is part of the thread dump (the complete thread dump will be attached to this jira)

      "Remoting "beve-7719" read-1" prio=5 tid=00000000028cc800 nid=0xb56ab000 waiting for monitor entry [00000000b56aa000]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at org.xnio.streams.BufferPipeOutputStream.close(BufferPipeOutputStream.java:156)
    - waiting to lock <0000000005697300> (a org.xnio.streams.BufferPipeOutputStream)
    at org.xnio.IoUtils.safeClose(IoUtils.java:137)
    at org.jboss.remoting3.remote.OutboundMessage.closeAsync(OutboundMessage.java:158)
    at org.jboss.remoting3.remote.RemoteConnectionChannel.handleAsyncClose(RemoteConnectionChannel.java:502)
    at org.jboss.remoting3.remote.RemoteReadListener.handleEvent(RemoteReadListener.java:259)
    at org.jboss.remoting3.remote.RemoteReadListener.handleEvent(RemoteReadListener.java:45)
    at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:72)
    at org.xnio.channels.TranslatingSuspendableChannel.handleReadable(TranslatingSuspendableChannel.java:189)
    at org.xnio.channels.TranslatingSuspendableChannel$1.handleEvent(TranslatingSuspendableChannel.java:103)
    at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:72)
    at org.xnio.nio.NioHandle.run(NioHandle.java:90)
    at org.xnio.nio.WorkerThread.run(WorkerThread.java:184)

   Locked ownable synchronizers:
    - None 

"EJB default - 4" prio=5 tid=0000000002a10c00 nid=0xb3971000 in Object.wait() [00000000b3970000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <00000000056972d0> (a org.jboss.remoting3.remote.OutboundMessage)
    at java.lang.Object.wait(Object.java:485)
    at org.jboss.remoting3.remote.OutboundMessage$1.accept(OutboundMessage.java:91)
    - locked <00000000056972d0> (a org.jboss.remoting3.remote.OutboundMessage)
    at org.xnio.streams.BufferPipeOutputStream.send(BufferPipeOutputStream.java:125)
    at org.xnio.streams.BufferPipeOutputStream.send(BufferPipeOutputStream.java:113)
    at org.xnio.streams.BufferPipeOutputStream.getBuffer(BufferPipeOutputStream.java:77)
    at org.xnio.streams.BufferPipeOutputStream.write(BufferPipeOutputStream.java:86)
    - locked <0000000005697300> (a org.xnio.streams.BufferPipeOutputStream)
    at org.jboss.remoting3.remote.OutboundMessage.write(OutboundMessage.java:168)
    at java.io.DataOutputStream.write(DataOutputStream.java:71)
    - locked <00000000056972b8> (a java.io.DataOutputStream)
    at org.jboss.as.ejb3.remote.protocol.versionone.AbstractMessageHandler$1.write(AbstractMessageHandler.java:188)
    at java.io.OutputStream.write(OutputStream.java:99)
    at org.jboss.marshalling.OutputStreamByteOutput.write(OutputStreamByteOutput.java:56)
    at org.jboss.marshalling.SimpleDataOutput.flush(SimpleDataOutput.java:311)
    at org.jboss.marshalling.SimpleDataOutput.write(SimpleDataOutput.java:82)
    at org.jboss.marshalling.river.BlockMarshaller.flush(BlockMarshaller.java:306)
    at org.jboss.marshalling.river.RiverMarshaller.writeEndBlock(RiverMarshaller.java:961)
    at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1008)
    at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:885)
    at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:825)
    at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1063)
    at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1019)
    at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:885)
    at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)
    at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119)
    at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.writeMethodInvocationResponse(MethodInvocationMessageHandler.java:338)
    at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.access$500(MethodInvocationMessageHandler.java:64)
    at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler$1.run(MethodInvocationMessageHandler.java:226)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:680)
    at org.jboss.threads.JBossThread.run(JBossThread.java:122)

   Locked ownable synchronizers:
    - <0000000008350f48> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

        1. ejb-deadlock.jar
          672 kB
        2. ejb-deadlock-maven.jar
          675 kB
        3. thread-dump.txt
          80 kB

              dlloyd@redhat.com David Lloyd
              dbeveniu Daniel Bevenius (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Created:
                Updated:
                Resolved: