Uploaded image for project: 'Red Hat Fuse'
  1. Red Hat Fuse
  2. ENTESB-11978

Gateway not closing connections on node

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Major
    • None
    • jboss-fuse-6.3
    • Gateway
    • None

    Description

      Fuse 6.3 R12,

      • Using gateway-mq and gateway-http on same container like the following architecture:
      JBossFuse:admin@root> container-list
      [id]        [version]  [type]  [connected]  [profiles]              [provision status]
      server1  1.0        karaf   yes          fabric                  success     
                                                  profile1                         
      server2  1.0        karaf   yes          fabric                  success     
                                                  profile2                         
      root*       1.0        karaf   yes          fabric                  success     
                                                  fabric-ensemble-0000-1              
                                                  jboss-fuse-full                     
        balancer  1.0        karaf   yes          default                 success     
                                                  gateway-http                        
                                                  gateway-mq
      
      • Any incoming request to services exponsed in server1 or server2 are being redirected internally multiple times and connections is not being closed, causing "Too many open files" error:
      2019-09-20 01:11:51,097 | WARN  | entloop-thread-0 | DefaultChannelPipeline           | 146 - io.netty.common - 4.0.37.Final-redhat-2 | An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
      java.io.IOException: Too many open files
      	at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)[:1.8.0_161]
      	at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)[:1.8.0_161]
      	at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)[:1.8.0_161]
      	at io.netty.channel.socket.nio.NioServerSocketChannel.doReadMessages(NioServerSocketChannel.java:135)[148:io.netty.transport:4.0.37.Final-redhat-2]
      	at io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:69)[148:io.netty.transport:4.0.37.Final-redhat-2]
      	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:564)[148:io.netty.transport:4.0.37.Final-redhat-2]
      	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:505)[148:io.netty.transport:4.0.37.Final-redhat-2]
      	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:419)[148:io.netty.transport:4.0.37.Final-redhat-2]
      	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:391)[148:io.netty.transport:4.0.37.Final-redhat-2]
      	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)[146:io.netty.common:4.0.37.Final-redhat-2]
      	at java.lang.Thread.run(Thread.java:748)[:1.8.0_161]
      2019-09-20 01:11:51,099 | INFO  | entloop-thread-0 | HttpGatewayHandler               | 139 - io.fabric8.gateway-core - 1.2.0.redhat-630187 | Proxying request /cxf/example/api/test/ping/1 to service path: /cxf/example/api/test/ping/1/ on service: http://server2:8181/cxf/example reverseServiceUrl: http://0.0.0.0:8060/cxf/example
      2019-09-20 01:11:51,101 | ERROR | entloop-thread-0 | DefaultContext                   | 138 - io.fabric8.fabric-vertx - 1.2.0.redhat-630187 | Unhandled exception
      io.netty.channel.ChannelException: Unable to create Channel from class class io.netty.channel.socket.nio.NioSocketChannel
      	at io.netty.bootstrap.AbstractBootstrap$BootstrapChannelFactory.newChannel(AbstractBootstrap.java:457)[148:io.netty.transport:4.0.37.Final-redhat-2]
      	at io.netty.bootstrap.AbstractBootstrap.initAndRegister(AbstractBootstrap.java:308)[148:io.netty.transport:4.0.37.Final-redhat-2]
      	at io.netty.bootstrap.Bootstrap.doConnect(Bootstrap.java:135)[148:io.netty.transport:4.0.37.Final-redhat-2]
      
      • netstat output shows connections on custom gateway-http port "8060" and random 47XX ports being increased and not closed/released:
      $ cat lsof_output.txt | grep -i 'balancerIP:8060'| wc -l
       2855703
      

      Attachments

        Issue Links

          Activity

            People

              ggrzybek Grzegorz Grzybek
              emunoz@redhat.com Elkin Dario Munoz Duarte
              Vratislav Hais Vratislav Hais (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: