Uploaded image for project: 'JBoss Enterprise Application Platform'
  1. JBoss Enterprise Application Platform
  2. JBEAP-2017

[GSS](7.0.z) Endpoint can be closed before doConnect tasks finished causing AbstractHandleableCloseable.close to wait forever

    XMLWordPrintable

Details

    • EAP 7.0.3

    Description

      Endpoint can be closed before doConnect tasks finished causing AbstractHandleableCloseable.close to wait forever.

      Seen when application is making remote naming calls to a remote server.

      Such as this:

              Properties props = new Properties();
        props.put("java.naming.factory.initial", "org.jboss.naming.remote.client.InitialContextFactory");
        ...
        Context remoteContext = null; 
        Object object = null;
        try {
           remoteContext = new InitialContext(props);
           object = remoteContext.lookup(lookup);
        } finally {
           remoteContext.close();
        }
        ...

      With cpu starvation under load, the timing of the connection start up tasks can be delayed. so JBoss received a connection before the endpoint close, but the full connection start (handleDone) didn't occur until after the endpoint close was initiated. This gets us into a state where the connection is able to come in, endpoint closes, then the connection finishes adding itself to the closed endpoint's connections map and will never be closed.

      EndpointImpl.doConnect

      • connHandlerFuture.getIoFuture().addNotifier(...
      • public void handleDone

      EndpointImpl.closeAction()

      https://github.com/jboss-remoting/jboss-remoting/blob/3.3.6.Final/src/main/java/org/jboss/remoting3/EndpointImpl.java#L277

      "Remoting "master:client-one:MANAGEMENT" task-3" prio=10 tid=0x00007f4a100fc800 nid=0x44b3 in Object.wait() [0x00007f4a68999000]
   java.lang.Thread.State: WAITING (on object monitor)
  at java.lang.Object.wait(Native Method) 
  - waiting on <0x00000000fe87aee0> (a java.lang.Object)
  at java.lang.Object.wait(Object.java:485)
  at org.jboss.remoting3.spi.AbstractHandleableCloseable.close(AbstractHandleableCloseable.java:190)
  - locked <0x00000000fe87aee0> (a java.lang.Object)
  at org.xnio.IoUtils.safeClose(IoUtils.java:137)
  at org.jboss.remoting3.EndpointImpl$5.handleDone(EndpointImpl.java:317)
  at org.jboss.remoting3.EndpointImpl$5.handleDone(EndpointImpl.java:293)
  at org.xnio.IoFuture$HandlingNotifier.notify(IoFuture.java:212)
  at org.xnio.AbstractIoFuture$NotifierRunnable.run(AbstractIoFuture.java:729)
  at org.xnio.IoUtils$2.execute(IoUtils.java:72)
  at org.xnio.AbstractIoFuture.runNotifier(AbstractIoFuture.java:702)
  at org.xnio.AbstractIoFuture$NotifierState.doNotify(AbstractIoFuture.java:275)
  at org.xnio.AbstractIoFuture$NotifierState.notifyDone(AbstractIoFuture.java:256)
  at org.xnio.AbstractIoFuture.setResult(AbstractIoFuture.java:628)
  at org.xnio.FutureResult.setResult(FutureResult.java:83)
  at org.xnio.IoUtils$ResultNotifier.handleDone(IoUtils.java:684)
  at org.xnio.IoUtils$ResultNotifier.handleDone(IoUtils.java:673)
  at org.xnio.IoFuture$HandlingNotifier.notify(IoFuture.java:212)
  at org.xnio.AbstractIoFuture$NotifierRunnable.run(AbstractIoFuture.java:729)
  at org.xnio.IoUtils$2.execute(IoUtils.java:72)
  at org.xnio.AbstractIoFuture.runNotifier(AbstractIoFuture.java:702)
  at org.xnio.AbstractIoFuture$NotifierState.doNotify(AbstractIoFuture.java:275)
  at org.xnio.AbstractIoFuture$NotifierState.notifyDone(AbstractIoFuture.java:256)
  at org.xnio.AbstractIoFuture$NotifierState.notifyDone(AbstractIoFuture.java:257)
  at org.xnio.AbstractIoFuture$CancellableState.notifyDone(AbstractIoFuture.java:334)
  at org.xnio.AbstractIoFuture.setResult(AbstractIoFuture.java:628)
  at org.xnio.FutureResult.setResult(FutureResult.java:83)
  at org.jboss.remoting3.remote.ClientConnectionOpenListener$Authentication$2.run(ClientConnectionOpenListener.java:763)
  at org.jboss.remoting3.EndpointImpl$TrackingExecutor$1.run(EndpointImpl.java:731)
  at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
  at java.lang.Thread.run(Thread.java:662)

      Attachments

        Issue Links

          clones

          Bug - A problem that impairs or prevents the functions of the product. JBEAP-5295 (7.1.0) Endpoint can be closed before doConnect tasks finished causing AbstractHandleableCloseable.close to wait forever

          • Critical - To be worked on immediately following BLOCKER issues.
          • Verified

          Bug - A problem that impairs or prevents the functions of the product. REM3-216 Endpoint can be closed before doConnect tasks finished causing AbstractHandleableCloseable.close to wait forever

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Resolved
          is caused by

          Bug - A problem that impairs or prevents the functions of the product. REM3-216 Endpoint can be closed before doConnect tasks finished causing AbstractHandleableCloseable.close to wait forever

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Resolved
          is incorporated by

          Component Upgrade - A task tracking an update to a bundled component or revision of component upstream of the project. JBEAP-5504 (7.0.z) Upgrade xnio from 3.3.6 to 3.4.0

          • Blocker - To be worked above all other priorities. This term is chosen when the severity of the issue is very high, has no workaround, or the effort for the change is comparatively low. Issues that may be very publicly visible and could generate significant media attention may also drive a higher priority.
          • Verified

          Component Upgrade - A task tracking an update to a bundled component or revision of component upstream of the project. JBEAP-3825 [GSS](7.0.z) Upgrade JBoss Remoting from 4.0.18 to 4.0.21

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Verified
          relates to

          Bug - A problem that impairs or prevents the functions of the product. REM3-221 Timeout errors in JCA bits in testsuite after JBEAP-2017

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Resolved

          Activity

            People

              elguardian@gmail.com Enrique González Martínez (Inactive)
              elguardian@gmail.com Enrique González Martínez (Inactive)
              Ladislav Thon Ladislav Thon
              Ladislav Thon Ladislav Thon
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: