Uploaded image for project: 'Undertow'
  1. Undertow
  2. UNDERTOW-676

mod_cluster proxy: Intermittent HTTP 503 on jvmKill based failover among worker nodes

XMLWordPrintable

      Correct pingpong with worker jboss-eap-7.0-3

      Note the worker node jboss-eap-7.0-3 that is about to get killed.

      06:56:17,603 DEBUG [io.undertow] (default I/O-11) UT005055: HttpClientPingTask run for connection: ajp://[2620:52:0:105f:0:0:ffff:50]:8009/?#
06:56:17,625 DEBUG [io.undertow] (default I/O-7) UT005055: HttpClientPingTask run for connection: ajp://[2620:52:0:105f:0:0:ffff:50]:8110/?#
06:56:17,706 DEBUG [io.undertow] (default I/O-43) UT005055: HttpClientPingTask run for connection: ajp://[2620:52:0:105f:0:0:ffff:50]:8211/?#
06:56:17,710 DEBUG [io.undertow] (default task-32) UT005056: Received node load in STATUS message, node jvmRoute: jboss-eap-7.0-2, load: 98
06:56:17,711 DEBUG [io.undertow] (default I/O-7) UT005057: Sending MCMP response to destination: /2620:52:0:105f:0:0:ffff:50:54598, HTTP status: 200, Headers: {X-Powered-By=[Undertow/1], Server=[JBoss-EAP/7], Content-Type=[text/plain; charset=ISO-8859-1]}, response: Type=STATUS-RSP&State=OK&JVMRoute=jboss-eap-7.0-2&id=1459767277573
06:56:17,740 DEBUG [io.undertow] (default task-33) UT005056: Received node load in STATUS message, node jvmRoute: jboss-eap-7.0-1, load: 98
06:56:17,741 DEBUG [io.undertow] (default I/O-11) UT005057: Sending MCMP response to destination: /2620:52:0:105f:0:0:ffff:50:54594, HTTP status: 200, Headers: {X-Powered-By=[Undertow/1], Server=[JBoss-EAP/7], Content-Type=[text/plain; charset=ISO-8859-1]}, response: Type=STATUS-RSP&State=OK&JVMRoute=jboss-eap-7.0-1&id=1459767277573
06:56:17,760 DEBUG [io.undertow] (default task-34) UT005056: Received node load in STATUS message, node jvmRoute: jboss-eap-7.0-3, load: 98
06:56:17,761 DEBUG [io.undertow] (default I/O-43) UT005057: Sending MCMP response to destination: /2620:52:0:105f:0:0:ffff:50:54610, HTTP status: 200, Headers: {X-Powered-By=[Undertow/1], Server=[JBoss-EAP/7], Content-Type=[text/plain; charset=ISO-8859-1]}, response: Type=STATUS-RSP&State=OK&JVMRoute=jboss-eap-7.0-3&id=1459767277573

      6:56:19,912 DEBUG [io.undertow.server.handlers.proxy] (default I/O-4) Sending request ClientRequest{path='/clusterbench/jvmroute', method=GET, protocol=HTTP/1.1} to target 2620:52:0:105f:0:0:ffff:50 for exchange HttpServerExchange{ GET /clusterbench/jvmroute request {Accept=[image/gif, image/jpeg, image/pjpeg, image/pjpeg, */*], Connection=[Keep-Alive], Accept-Language=[en-us], Accept-Encoding=[gzip, deflate], User-Agent=[Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0)], Host=[[2620:52:0:105f::ffff:50]:8484]} response {X-Powered-By=[Undertow/1], Server=[JBoss-EAP/7]}}
6:56:19,914 DEBUG [io.undertow.server.handlers.proxy] (default I/O-4) Sent request ClientRequest{path='/clusterbench/jvmroute', method=GET, protocol=HTTP/1.1} to target 2620:52:0:105f:0:0:ffff:50 for exchange HttpServerExchange{ GET /clusterbench/jvmroute request {Accept=[image/gif, image/jpeg, image/pjpeg, image/pjpeg, */*], Connection=[Keep-Alive], Accept-Language=[en-us], Accept-Encoding=[gzip, deflate], User-Agent=[Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0)], Host=[[2620:52:0:105f::ffff:50]:8484]} response {X-Powered-By=[Undertow/1], Server=[JBoss-EAP/7]}}
6:56:19,917 DEBUG [io.undertow.request] (default I/O-4) suspending writes on io.undertow.protocols.ajp.AjpClientRequestClientStreamSinkChannel@4ddb9aa5 to prevent listener runaway
6:56:19,963 DEBUG [io.undertow.server.handlers.proxy] (default I/O-4) Received response io.undertow.client.ClientResponse@3fe16b66 for request ClientRequest{path='/clusterbench/jvmroute', method=GET, protocol=HTTP/1.1} for exchange HttpServerExchange{ GET /clusterbench/jvmroute request {Accept=[image/gif, image/jpeg, image/pjpeg, image/pjpeg, */*], Connection=[Keep-Alive], Accept-Language=[en-us], Accept-Encoding=[gzip, deflate], User-Agent=[Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0)], Host=[[2620:52:0:105f::ffff:50]:8484]} response {X-Powered-By=[Undertow/1], Server=[JBoss-EAP/7]}}

      Killing the node

      Note the time:

      06:56:19.985 [INFO] --- Starting killing the server jboss-eap-7.0-3 ---

      Undertow tries to connect to jboss-eap-7.0-3

      IMHO, there should have been a new pingpong before entrusting dead jboss-eap-7.0-3 with the reqeust . The error is propagated to the client as HTTP 503 and that is unacceptable mod_clister-wise.

      06:56:20,067 DEBUG [io.undertow.request] (default I/O-4) Failed to connect: java.net.ConnectException: Connection refused
    at sun.nio.ch.Net.connect0(Native Method)
    at sun.nio.ch.Net.connect(Net.java:454)
    at sun.nio.ch.Net.connect(Net.java:446)
    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648)
    at org.xnio.nio.WorkerThread.openTcpStreamConnection(WorkerThread.java:264)
    at org.xnio.XnioIoThread.openStreamConnection(XnioIoThread.java:224)
    at io.undertow.client.ajp.AjpClientProvider.connect(AjpClientProvider.java:106)
    at io.undertow.client.UndertowClient.connect(UndertowClient.java:157)
    at io.undertow.server.handlers.proxy.ProxyConnectionPool.openConnection(ProxyConnectionPool.java:265)
    at io.undertow.server.handlers.proxy.ProxyConnectionPool.connect(ProxyConnectionPool.java:514)
    at io.undertow.server.handlers.proxy.mod_cluster.Context.handleRequest(Context.java:179)
    at io.undertow.server.handlers.proxy.mod_cluster.ModClusterProxyClient.getConnection(ModClusterProxyClient.java:124)
    at io.undertow.server.handlers.proxy.ProxyHandler$ProxyClientHandler.run(ProxyHandler.java:264)
    at io.undertow.util.SameThreadExecutor.execute(SameThreadExecutor.java:35)
    at io.undertow.server.Connectors.executeRootHandler(Connectors.java:215)
    at io.undertow.server.protocol.http.HttpReadListener.handleEventWithNoRunningRequest(HttpReadListener.java:233)
    at io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:131)
    at io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:57)
    at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
    at org.xnio.conduits.ReadReadyHandler$ChannelListenerHandler.readReady(ReadReadyHandler.java:66)
    at org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:88)
    at org.xnio.nio.WorkerThread.run(WorkerThread.java:559)

06:56:20,067 DEBUG [io.undertow] (default I/O-4) UT005048: Node jboss-eap-7.0-3 in error

      Configuration

      Default, i.e.

      <subsystem xmlns="urn:jboss:domain:undertow:3.1">
    <buffer-cache name="default"/>
    <server name="default-server">
        <http-listener name="default" socket-binding="http" redirect-socket="https"/>
        <host name="default-host" alias="localhost">
            <location name="/" handler="welcome-content"/>
            <filter-ref name="server-header"/>
            <filter-ref name="x-powered-by-header"/>
            <filter-ref name="modcluster"/>
        </host>
    </server>
    <servlet-container name="default">
        <jsp-config/>
        <websockets/>
    </servlet-container>
    <handlers>
        <file name="welcome-content" path="${jboss.home.dir}/welcome-content"/>
    </handlers>
    <filters>
        <response-header name="server-header" header-name="Server" header-value="JBoss-EAP/7"/>
        <response-header name="x-powered-by-header" header-name="X-Powered-By" header-value="Undertow/1"/>
        <mod-cluster name="modcluster" advertise-socket-binding="modcluster-adv" management-socket-binding="http"/>
    </filters>
</subsystem>

      WDYT?

              sdouglas1@redhat.com Stuart Douglas (Inactive)
              mbabacek1@redhat.com Karm Karm
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: