Details
-
Bug
-
Resolution: Done
-
Critical
-
jboss-fuse-6.2.1
-
%
Description
The structure of fabric environment is as below:
[id] [version] [profiles]
root 1.0 fabric fabric-ensemble-0001-1
child 1.0 app
childhttpgw 1.0 gateway-http
root2 1.0 fabric fabric-ensemble-0001-2
child2 1.0 app
childhttpgw2 1.0 gateway-http
root3 1.0 fabric fabric-ensemble-0001-3
child3 1.0 app
Normally, a load balancer sends requests to gateway-http containers, then it will be forwarded to worker containers.
But RANDOMLY, gateway-http container will throw below error:
2017-04-13 19:32:49,921 | INFO | entloop-thread-0 | HttpGatewayHandler | 110 - io.fabric8.gateway-core - 1.2.0.redhat-621107 | Proxying request /cxf/blackListService to service path: /cxf/blackListService/ on service: http://abc:9090/cxf/blackListService reverseServiceUrl: http://0.0.0.0:9000/cxf/blackListService 2017-04-13 19:33:30,546 | INFO | entloop-thread-0 | HttpGatewayHandler | 110 - io.fabric8.gateway-core - 1.2.0.redhat-621107 | Proxying request /cxf/blackListService to service path: /cxf/blackListService/ on service: http://abc:9090/cxf/blackListService reverseServiceUrl: http://0.0.0.0:9000/cxf/blackListService 2017-04-13 19:33:51,484 | INFO | entloop-thread-0 | HttpGatewayHandler | 110 - io.fabric8.gateway-core - 1.2.0.redhat-621107 | Proxying request /cxf/blackListService to service path: /cxf/blackListService/ on service: http://abc:-1/cxf/blackListService reverseServiceUrl: http://0.0.0.0:9000/cxf/blackListService 2017-04-13 19:33:51,484 | ERROR | entloop-thread-0 | DefaultContext | 109 - io.fabric8.fabric-vertx - 1.2.0.redhat-621107 | Unhandled exception java.lang.IllegalArgumentException: port out of range:-1 at java.net.InetSocketAddress.checkPort(InetSocketAddress.java:143)[:1.8.0_91] at java.net.InetSocketAddress.<init>(InetSocketAddress.java:224)[:1.8.0_91] at org.vertx.java.core.http.impl.DefaultHttpClient.internalConnect(DefaultHttpClient.java:737)[109:io.fabric8.fabric-vertx:1.2.0.redhat-621107] at org.vertx.java.core.http.impl.DefaultHttpClient$1.connect(DefaultHttpClient.java:68)[109:io.fabric8.fabric-vertx:1.2.0.redhat-621107] at org.vertx.java.core.http.impl.PriorityHttpConnectionPool.getConnection(PriorityHttpConnectionPool.java:103)[109:io.fabric8.fabric-vertx:1.2.0.redhat-621107] at org.vertx.java.core.http.impl.DefaultHttpClient.getConnection(DefaultHttpClient.java:678)[109:io.fabric8.fabric-vertx:1.2.0.redhat-621107] at org.vertx.java.core.http.impl.DefaultHttpClientRequest.connect(DefaultHttpClientRequest.java:327)[109:io.fabric8.fabric-vertx:1.2.0.redhat-621107] at org.vertx.java.core.http.impl.DefaultHttpClientRequest.write(DefaultHttpClientRequest.java:459)[109:io.fabric8.fabric-vertx:1.2.0.redhat-621107] at org.vertx.java.core.http.impl.DefaultHttpClientRequest.write(DefaultHttpClientRequest.java:118)[109:io.fabric8.fabric-vertx:1.2.0.redhat-621107] at org.vertx.java.core.http.impl.DefaultHttpClientRequest.write(DefaultHttpClientRequest.java:36)[109:io.fabric8.fabric-vertx:1.2.0.redhat-621107] at io.fabric8.gateway.handlers.http.HttpGatewayHandler$2.handle(HttpGatewayHandler.java:172)[110:io.fabric8.gateway-core:1.2.0.redhat-621107] at io.fabric8.gateway.handlers.http.HttpGatewayHandler$2.handle(HttpGatewayHandler.java:167)[110:io.fabric8.gateway-core:1.2.0.redhat-621107] at org.vertx.java.core.http.impl.DefaultHttpServerRequest.handleData(DefaultHttpServerRequest.java:284)[109:io.fabric8.fabric-vertx:1.2.0.redhat-621107] at org.vertx.java.core.http.impl.ServerConnection.handleChunk(ServerConnection.java:200)[109:io.fabric8.fabric-vertx:1.2.0.redhat-621107] at org.vertx.java.core.http.impl.ServerConnection.processMessage(ServerConnection.java:307)[109:io.fabric8.fabric-vertx:1.2.0.redhat-621107] at org.vertx.java.core.http.impl.ServerConnection.handleMessage(ServerConnection.java:94)[109:io.fabric8.fabric-vertx:1.2.0.redhat-621107] at org.vertx.java.core.http.impl.DefaultHttpServer$ServerHandler.doMessageReceived(DefaultHttpServer.java:703)[109:io.fabric8.fabric-vertx:1.2.0.redhat-621107] at org.vertx.java.core.http.impl.DefaultHttpServer$ServerHandler.doMessageReceived(DefaultHttpServer.java:593)[109:io.fabric8.fabric-vertx:1.2.0.redhat-621107] at org.vertx.java.core.http.impl.VertxHttpHandler.channelRead(VertxHttpHandler.java:72)[109:io.fabric8.fabric-vertx:1.2.0.redhat-621107] at org.vertx.java.core.net.impl.VertxHandler.channelRead(VertxHandler.java:155)[109:io.fabric8.fabric-vertx:1.2.0.redhat-621107] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)[119:io.netty.transport:4.0.27.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)[119:io.netty.transport:4.0.27.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242)[115:io.netty.codec:4.0.27.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)[119:io.netty.transport:4.0.27.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)[119:io.netty.transport:4.0.27.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847)[119:io.netty.transport:4.0.27.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)[119:io.netty.transport:4.0.27.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)[119:io.netty.transport:4.0.27.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)[119:io.netty.transport:4.0.27.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)[119:io.netty.transport:4.0.27.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)[119:io.netty.transport:4.0.27.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)[117:io.netty.common:4.0.27.Final] at java.lang.Thread.run(Thread.java:745)[:1.8.0_91]
Then the request will not be sent to worker container.
We can see at "2017-04-13 19:33:51,484" the port was -1, good request has port 9090, this happens sometimes.
From worker container, we can see weird info as below:
2017-04-13 18:01:32,781 | INFO | MCF-2-thread-1 | FabricCxfRegistrationHandler | 200 - io.fabric8.fabric-cxf-registry - 1.2.0.redhat-621107 | Registered CXF API at /fabric/registry/clusters/apis/rest/LOVService/lookup/1.0/child2 JSON: {"id":"child2", "container":"child2", "version":"1.0", "services":["http://${zk:child2/ip}:-1/cxf/blackListService"], "objectName":"io.fabric8.cxf:bus.id=child-lookup-service-cxf972756723,type=Bus.Service.Endpoint,service=\"{http://abc.com/}LOVService\",port=\"LOVService\",instance.id=1760657389", "wadl":"?_wadl"}
The port is "-1" as well, from command "zk:get" which confirmed it:
JBossFuse:fusewebadm@root> zk:get /fabric/registry/clusters/apis/rest/BlackListService/blackListService/1.0/child2{"id":"child2", "container":"child2", "version":"1.0", "services":["http://${zk:child2/ip}:-1/cxf/blackListService"],"objectName":"io.fabric8.cxf:bus.id=child-blacklist-service-cxf1677890329,type=Bus.Service.Endpoint,service=\"{http://abc.com/}BlackListService\",port=\"BlackListService\",instance.id=1925874623", "wadl":"?_wadl"}
The bluprint.xml:
<cxf:rsServer address="{{fuse.host.url}}/blackListService" id="blackListService" serviceClass="com.abc.BlackListService"> <cxf:providers> <bean class="com.fasterxml.jackson.jaxrs.json.JacksonJsonProvider"/> <bean class="com.abc.DefExceptionMapper" id="rsException"/> </cxf:providers> </cxf:rsServer> ----- fuse.host.url = http://FUSE_IP:9090/cxf
And there are serval services encountered this issue:
Registered CXF API ..."services":["http://${zk:child2/ip}:-1/cxf/lookup"] Registered CXF API ..."services":["http://${zk:child2/ip}:-1/cxf/premiaService"] Registered CXF API ..."services":["http://${zk:child2/ip}:-1/cxf/blackListService"]
Customer can observer this issue by using "Postman" as well, but didn't find the condition to trigger it.
We then did similar test and put load on it:
<cm:property-placeholder persistent-id="org.apache.servicemix.examples.cxf.receive" update-strategy="reload"> <cm:default-properties/> </cm:property-placeholder> <cxf:rsServer id="rsServer" address="{{CXFserver}}{{service}}" ... ---cxf container--- 2017-04-19 17:12:35,163 | INFO | MCF-1-thread-1 | FabricCxfRegistrationHandler | 117 - io.fabric8.fabric-cxf-registry - 1.2.0.redhat-621107 | Registered CXF API at /fabric/registry/clusters/apis/rest/PersonService/rest/1.0/cxf JSON: {"id":"cxf", "container":"cxf", "version":"1.0", "services":["http://${zk:cxf/ip}:8989/rest"], "objectName":"io.fabric8.cxf:bus.id=camel-cxf-rest-route-cxf1366049186,type=Bus.Service.Endpoint,service=\"{http://rest.camel.examples.servicemix.apache.org/}PersonService\",port=\"PersonService\",instance.id=922557029", "wadl":"?_wadl"} ------------------- When client visit gateway-http, it works fine: ---gateway container--- 2017-04-19 17:14:04,076 | INFO | entloop-thread-0 | HttpGatewayHandler | 99 - io.fabric8.gateway-core - 1.2.0.redhat-621107 | Proxying request /rest/personservice/person/delete/1 to service path: /rest/personservice/person/delete/1 on service: http://xuzhan.pek.csb:8989/rest reverseServiceUrl: http://0.0.0.0:9000/rest
We can see the port was registered correctly.
Right now, customer let load balancer send requests directly to worker containers to avoid this issue in production.
Attached the log and screen shot from customer.
