Uploaded image for project: 'JBoss Enterprise Application Platform'
  1. JBoss Enterprise Application Platform
  2. JBEAP-5295

(7.1.0) Endpoint can be closed before doConnect tasks finished causing AbstractHandleableCloseable.close to wait forever

    Details

      Description

      Endpoint can be closed before doConnect tasks finished causing AbstractHandleableCloseable.close to wait forever.

      Seen when application is making remote naming calls to a remote server.

      Such as this:

              Properties props = new Properties();
              props.put("java.naming.factory.initial", "org.jboss.naming.remote.client.InitialContextFactory");
              ...
              Context remoteContext = null; 
              Object object = null;
              try {
                 remoteContext = new InitialContext(props);
                 object = remoteContext.lookup(lookup);
              } finally {
                 remoteContext.close();
              }
              ...
      

      With cpu starvation under load, the timing of the connection start up tasks can be delayed. so JBoss received a connection before the endpoint close, but the full connection start (handleDone) didn't occur until after the endpoint close was initiated. This gets us into a state where the connection is able to come in, endpoint closes, then the connection finishes adding itself to the closed endpoint's connections map and will never be closed.

      EndpointImpl.doConnect

      • connHandlerFuture.getIoFuture().addNotifier(...
      • public void handleDone

      EndpointImpl.closeAction()

      https://github.com/jboss-remoting/jboss-remoting/blob/3.3.6.Final/src/main/java/org/jboss/remoting3/EndpointImpl.java#L277

      "Remoting "master:client-one:MANAGEMENT" task-3" prio=10 tid=0x00007f4a100fc800 nid=0x44b3 in Object.wait() [0x00007f4a68999000]
         java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method) 
        - waiting on <0x00000000fe87aee0> (a java.lang.Object)
        at java.lang.Object.wait(Object.java:485)
        at org.jboss.remoting3.spi.AbstractHandleableCloseable.close(AbstractHandleableCloseable.java:190)
        - locked <0x00000000fe87aee0> (a java.lang.Object)
        at org.xnio.IoUtils.safeClose(IoUtils.java:137)
        at org.jboss.remoting3.EndpointImpl$5.handleDone(EndpointImpl.java:317)
        at org.jboss.remoting3.EndpointImpl$5.handleDone(EndpointImpl.java:293)
        at org.xnio.IoFuture$HandlingNotifier.notify(IoFuture.java:212)
        at org.xnio.AbstractIoFuture$NotifierRunnable.run(AbstractIoFuture.java:729)
        at org.xnio.IoUtils$2.execute(IoUtils.java:72)
        at org.xnio.AbstractIoFuture.runNotifier(AbstractIoFuture.java:702)
        at org.xnio.AbstractIoFuture$NotifierState.doNotify(AbstractIoFuture.java:275)
        at org.xnio.AbstractIoFuture$NotifierState.notifyDone(AbstractIoFuture.java:256)
        at org.xnio.AbstractIoFuture.setResult(AbstractIoFuture.java:628)
        at org.xnio.FutureResult.setResult(FutureResult.java:83)
        at org.xnio.IoUtils$ResultNotifier.handleDone(IoUtils.java:684)
        at org.xnio.IoUtils$ResultNotifier.handleDone(IoUtils.java:673)
        at org.xnio.IoFuture$HandlingNotifier.notify(IoFuture.java:212)
        at org.xnio.AbstractIoFuture$NotifierRunnable.run(AbstractIoFuture.java:729)
        at org.xnio.IoUtils$2.execute(IoUtils.java:72)
        at org.xnio.AbstractIoFuture.runNotifier(AbstractIoFuture.java:702)
        at org.xnio.AbstractIoFuture$NotifierState.doNotify(AbstractIoFuture.java:275)
        at org.xnio.AbstractIoFuture$NotifierState.notifyDone(AbstractIoFuture.java:256)
        at org.xnio.AbstractIoFuture$NotifierState.notifyDone(AbstractIoFuture.java:257)
        at org.xnio.AbstractIoFuture$CancellableState.notifyDone(AbstractIoFuture.java:334)
        at org.xnio.AbstractIoFuture.setResult(AbstractIoFuture.java:628)
        at org.xnio.FutureResult.setResult(FutureResult.java:83)
        at org.jboss.remoting3.remote.ClientConnectionOpenListener$Authentication$2.run(ClientConnectionOpenListener.java:763)
        at org.jboss.remoting3.EndpointImpl$TrackingExecutor$1.run(EndpointImpl.java:731)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
        at java.lang.Thread.run(Thread.java:662)
      

        Gliffy Diagrams

          Attachments

            Issue Links

              Activity

                People

                • Assignee:
                  elguardian Enrique González Martínez
                  Reporter:
                  baranowb Bartosz Baranowski
                  Tester:
                  Richard Janik
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  6 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: