Uploaded image for project: 'Application Server 7'
  1. Application Server 7
  2. AS7-4274

Remoting deadlock after deserialization exception of a large object graph returned from EJB.

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Major
    • 7.1.2.Final (EAP)
    • 7.1.1.Final
    • EJB, Remoting
    • None
    • Hide

      Install/Unpack

      1. Unpack ejb-deadlock.jar to a local directory
      2. cd ejb-deadlock

      Deploy

      1. deploy target/lib/ejb-deadlock.jar to AS 7.1.1.Final

      Running the client

      The client se.rl.deadlock.CustomerBeanClient reproduces the issue by using Byteman to throw a fake ClassNotFoundException. The Byteman configuration can be found in src/test/resources/customer-bean-client.btm.

      The client can be run from the command line via Gradle:

      ./gradlew exec

      Or run from within an IDE, but requires setting the following VM argument:

      -javaagent:lib/byteman.jar=script:src/test/resources/customer-bean-client.btm

      The source will be attached but can also be found on github

      standalone/conf/standalone.xml

      Under

      <subsystem xmlns="urn:jboss:domain:remoting:1.1">

      Add

      <outbound-connections>
  <remote-outbound-connection name="remote-ejb-connection1" outbound-socket-binding-ref="remote-ejb">
  </remote-outbound-connection>
  <remote-outbound-connection name="remote-ejb-connection2" outbound-socket-binding-ref="remote-ejb">
  </remote-outbound-connection>
</outbound-connections>

      and

      Under

      <socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">

      Add

      <outbound-socket-binding name="remote-ejb">
  <remote-destination host="localhost" port="4447"/>
</outbound-socket-binding>
      Show
      Install/Unpack Unpack ejb-deadlock.jar to a local directory cd ejb-deadlock Deploy deploy target/lib/ejb-deadlock.jar to AS 7.1.1.Final Running the client The client se.rl.deadlock.CustomerBeanClient reproduces the issue by using Byteman to throw a fake ClassNotFoundException. The Byteman configuration can be found in src/test/resources/customer-bean-client.btm . The client can be run from the command line via Gradle: ./gradlew exec Or run from within an IDE, but requires setting the following VM argument: -javaagent:lib/byteman.jar=script:src/test/resources/customer-bean-client.btm The source will be attached but can also be found on github standalone/conf/standalone.xml Under <subsystem xmlns= "urn:jboss:domain:remoting:1.1" > Add <outbound-connections> <remote-outbound-connection name= "remote-ejb-connection1" outbound-socket-binding-ref= "remote-ejb" > </remote-outbound-connection> <remote-outbound-connection name= "remote-ejb-connection2" outbound-socket-binding-ref= "remote-ejb" > </remote-outbound-connection> </outbound-connections> and Under <socket-binding-group name= "standard-sockets" default - interface = " public " port-offset= "${jboss.socket.binding.port-offset:0}" > Add <outbound-socket-binding name= "remote-ejb" > <remote-destination host= "localhost" port= "4447" /> </outbound-socket-binding>

    Description

      The use case is a remote EJB client that calls an EJB which returns a large object graph. Due to a configuration error on our part a class that is part of this object graph is not available in the clients classpath. This causes a ClassNotFoundException to be thrown when deserializing, and after this exception is thrown the client "freezes" but can be interrupted by CTRL+C, but the server will deadlock and will not respond and has to be shutdown by killing the server process.

      Below is part of the thread dump (the complete thread dump will be attached to this jira)

      "Remoting "beve-7719" read-1" prio=5 tid=00000000028cc800 nid=0xb56ab000 waiting for monitor entry [00000000b56aa000]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at org.xnio.streams.BufferPipeOutputStream.close(BufferPipeOutputStream.java:156)
    - waiting to lock <0000000005697300> (a org.xnio.streams.BufferPipeOutputStream)
    at org.xnio.IoUtils.safeClose(IoUtils.java:137)
    at org.jboss.remoting3.remote.OutboundMessage.closeAsync(OutboundMessage.java:158)
    at org.jboss.remoting3.remote.RemoteConnectionChannel.handleAsyncClose(RemoteConnectionChannel.java:502)
    at org.jboss.remoting3.remote.RemoteReadListener.handleEvent(RemoteReadListener.java:259)
    at org.jboss.remoting3.remote.RemoteReadListener.handleEvent(RemoteReadListener.java:45)
    at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:72)
    at org.xnio.channels.TranslatingSuspendableChannel.handleReadable(TranslatingSuspendableChannel.java:189)
    at org.xnio.channels.TranslatingSuspendableChannel$1.handleEvent(TranslatingSuspendableChannel.java:103)
    at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:72)
    at org.xnio.nio.NioHandle.run(NioHandle.java:90)
    at org.xnio.nio.WorkerThread.run(WorkerThread.java:184)

   Locked ownable synchronizers:
    - None 

"EJB default - 4" prio=5 tid=0000000002a10c00 nid=0xb3971000 in Object.wait() [00000000b3970000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <00000000056972d0> (a org.jboss.remoting3.remote.OutboundMessage)
    at java.lang.Object.wait(Object.java:485)
    at org.jboss.remoting3.remote.OutboundMessage$1.accept(OutboundMessage.java:91)
    - locked <00000000056972d0> (a org.jboss.remoting3.remote.OutboundMessage)
    at org.xnio.streams.BufferPipeOutputStream.send(BufferPipeOutputStream.java:125)
    at org.xnio.streams.BufferPipeOutputStream.send(BufferPipeOutputStream.java:113)
    at org.xnio.streams.BufferPipeOutputStream.getBuffer(BufferPipeOutputStream.java:77)
    at org.xnio.streams.BufferPipeOutputStream.write(BufferPipeOutputStream.java:86)
    - locked <0000000005697300> (a org.xnio.streams.BufferPipeOutputStream)
    at org.jboss.remoting3.remote.OutboundMessage.write(OutboundMessage.java:168)
    at java.io.DataOutputStream.write(DataOutputStream.java:71)
    - locked <00000000056972b8> (a java.io.DataOutputStream)
    at org.jboss.as.ejb3.remote.protocol.versionone.AbstractMessageHandler$1.write(AbstractMessageHandler.java:188)
    at java.io.OutputStream.write(OutputStream.java:99)
    at org.jboss.marshalling.OutputStreamByteOutput.write(OutputStreamByteOutput.java:56)
    at org.jboss.marshalling.SimpleDataOutput.flush(SimpleDataOutput.java:311)
    at org.jboss.marshalling.SimpleDataOutput.write(SimpleDataOutput.java:82)
    at org.jboss.marshalling.river.BlockMarshaller.flush(BlockMarshaller.java:306)
    at org.jboss.marshalling.river.RiverMarshaller.writeEndBlock(RiverMarshaller.java:961)
    at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1008)
    at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:885)
    at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:825)
    at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1063)
    at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1019)
    at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:885)
    at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)
    at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119)
    at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.writeMethodInvocationResponse(MethodInvocationMessageHandler.java:338)
    at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.access$500(MethodInvocationMessageHandler.java:64)
    at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler$1.run(MethodInvocationMessageHandler.java:226)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:680)
    at org.jboss.threads.JBossThread.run(JBossThread.java:122)

   Locked ownable synchronizers:
    - <0000000008350f48> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

      Attachments

        1. thread-dump.txt
          80 kB
        2. ejb-deadlock-maven.jar
          675 kB
        3. ejb-deadlock.jar
          672 kB

        Issue Links

          is blocked by

          Bug - A problem that impairs or prevents the functions of the product. REM3-148 Remoting hangs if the client closes a connection when the server is still sending message(s) on the channel for that connection

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Resolved
          relates to

          Bug - A problem that impairs or prevents the functions of the product. AS7-5592 Regression: EJB remote invocations hang indefinitely due to possible remoting issues

          • Critical - To be worked on immediately following BLOCKER issues.
          • Resolved

          Activity

            People

              dlloyd@redhat.com David Lloyd
              dbeveniu Daniel Bevenius (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: