Details
-
Bug
-
Resolution: Done
-
Blocker
-
jboss-fuse-6.2
-
None
-
%
-
Sprint 5 - towards ER2
Description
An installation has several fabric gateways distributing requests to ActiveMQ message brokers in fabric instances. There is a further, upstream load balancer ahead of the gateways.
Under load, after some time, one of the gateways stops accepting further requests. The problem is first noticed in an exception handler, although the actual exception is never reported. The exception handler tries to call release() on the gateway's ShutdownHandler, but this call fails with the following exception:
2015-12-01 23:10:54,165 | ERROR | entloop-thread-2 | DefaultContext | 266 - io.fabric8.fabric-vertx - 1.2.0.redhat-133 | Unhandled exception java.lang.IllegalStateException: Unbalanced calls to release detected. at io.fabric8.common.util.ShutdownTracker.release(ShutdownTracker.java:78)[65:io.fabric8.common-util:1.2.0.redhat-133] at io.fabric8.gateway.handlers.detecting.DetectingGateway.handleShutdown(DetectingGateway.java:430)[268:io.fabric8.gateway-core:1.2.0.redhat-133] at io.fabric8.gateway.handlers.detecting.DetectingGateway.access$400(DetectingGateway.java:54)[268:io.fabric8.gateway-core:1.2.0.redhat-133] at io.fabric8.gateway.handlers.detecting.DetectingGateway$6$2.handle(DetectingGateway.java:410)[268:io.fabric8.gateway-core:1.2.0.redhat-133] at io.fabric8.gateway.handlers.detecting.DetectingGateway$6$2.handle(DetectingGateway.java:407)[268:io.fabric8.gateway-core:1.2.0.redhat-133] at org.vertx.java.core.net.impl.ConnectionBase.handleException(ConnectionBase.java:126)[266:io.fabric8.fabric-vertx:1.2.0.redhat-133] at org.vertx.java.core.net.impl.VertxHandler$2.run(VertxHandler.java:104)[266:io.fabric8.fabric-vertx:1.2.0.redhat-133] at org.vertx.java.core.impl.DefaultContext$3.run(DefaultContext.java:175)[266:io.fabric8.fabric-vertx:1.2.0.redhat-133] at org.vertx.java.core.impl.DefaultContext.execute(DefaultContext.java:135)[266:io.fabric8.fabric-vertx:1.2.0.redhat-133] at org.vertx.java.core.net.impl.VertxHandler.exceptionCaught(VertxHandler.java:96)[266:io.fabric8.fabric-vertx:1.2.0.redhat-133] at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:275)[277:io.netty.transport:4.0.27.Final] at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:253)[277:io.netty.transport:4.0.27.Final] at io.netty.channel.DefaultChannelPipeline.fireExceptionCaught(DefaultChannelPipeline.java:835)[277:io.netty.transport:4.0.27.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.handleReadException(AbstractNioByteChannel.java:87)[277:io.netty.transport:4.0.27.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:162)[277:io.netty.transport:4.0.27.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)[277:io.netty.transport:4.0.27.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)[277:io.netty.transport:4.0.27.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)[277:io.netty.transport:4.0.27.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)[277:io.netty.transport:4.0.27.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)[275:io.netty.common:4.0.27.Final] at java.lang.Thread.run(Thread.java:745)[:1.8.0_65]
Thereafter the gateway is in a peculiar state, and all further requests on it fail, with the following exception:
2015-12-01 23:10:54,413 | ERROR | entloop-thread-2 | DefaultContext | 266 - io.fabric8.fabric-vertx - 1.2.0.redhat-133 | Unhandled exception io.fabric8.common.util.ShutdownTracker$ShutdownException at io.fabric8.common.util.ShutdownTracker.retain(ShutdownTracker.java:51)[65:io.fabric8.common-util:1.2.0.redhat-133] at io.fabric8.gateway.handlers.detecting.DetectingGateway.handle(DetectingGateway.java:206)[268:io.fabric8.gateway-core:1.2.0.redhat-133] at io.fabric8.gateway.handlers.detecting.DetectingGatewayNetSocketHandler.handle(DetectingGatewayNetSocketHandler.java:33)[268:io.fabric8.gateway-core:1.2.0.redhat-133] at io.fabric8.gateway.handlers.detecting.DetectingGatewayNetSocketHandler.handle(DetectingGatewayNetSocketHandler.java:24)[268:io.fabric8.gateway-core:1.2.0.redhat-133] at org.vertx.java.core.net.impl.DefaultNetServer$ServerHandler.doConnected(DefaultNetServer.java:572)[266:io.fabric8.fabric-vertx:1.2.0.redhat-133] at org.vertx.java.core.net.impl.DefaultNetServer$ServerHandler.access$1400(DefaultNetServer.java:525)[266:io.fabric8.fabric-vertx:1.2.0.redhat-133] at org.vertx.java.core.net.impl.DefaultNetServer$ServerHandler$2.run(DefaultNetServer.java:564)[266:io.fabric8.fabric-vertx:1.2.0.redhat-133] at org.vertx.java.core.impl.DefaultContext$3.run(DefaultContext.java:175)[266:io.fabric8.fabric-vertx:1.2.0.redhat-133] at org.vertx.java.core.impl.DefaultContext.execute(DefaultContext.java:135)[266:io.fabric8.fabric-vertx:1.2.0.redhat-133] at org.vertx.java.core.net.impl.DefaultNetServer$ServerHandler.connected(DefaultNetServer.java:562)[266:io.fabric8.fabric-vertx:1.2.0.redhat-133] at org.vertx.java.core.net.impl.DefaultNetServer$ServerHandler.channelActive(DefaultNetServer.java:557)[266:io.fabric8.fabric-vertx:1.2.0.redhat-133] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:212)[277:io.netty.transport:4.0.27.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:198)[277:io.netty.transport:4.0.27.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelActive(DefaultChannelPipeline.java:818)[277:io.netty.transport:4.0.27.Final] at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:454)[277:io.netty.transport:4.0.27.Final] at io.netty.channel.AbstractChannel$AbstractUnsafe.register(AbstractChannel.java:418)[277:io.netty.transport:4.0.27.Final] at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:60)[277:io.netty.transport:4.0.27.Final] at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:48)[277:io.netty.transport:4.0.27.Final] at org.vertx.java.core.net.impl.VertxEventLoopGroup.register(VertxEventLoopGroup.java:62)[266:io.fabric8.fabric-vertx:1.2.0.redhat-133] at io.netty.bootstrap.ServerBootstrap$ServerBootstrapAcceptor.channelRead(ServerBootstrap.java:251)[277:io.netty.transport:4.0.27.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)[277:io.netty.transport:4.0.27.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)[277:io.netty.transport:4.0.27.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847)[277:io.netty.transport:4.0.27.Final] at io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:93)[277:io.netty.transport:4.0.27.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)[277:io.netty.transport:4.0.27.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)[277:io.netty.transport:4.0.27.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)[277:io.netty.transport:4.0.27.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)[277:io.netty.transport:4.0.27.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)[275:io.netty.common:4.0.27.Final] at java.lang.Thread.run(Thread.java:745)[:1.8.0_65]
It seems that in some cases we end up in a situation where an operation that shut only be performed during shutdown is performed in response to an exception, and that leaves the gateway in an indeterminate state.
