Uploaded image for project: 'JBoss Enterprise Application Platform'
  1. JBoss Enterprise Application Platform
  2. JBEAP-14141

[GSS](7.1.z) MODCLUSTER-639 - proxy reset requests can allow for other MCMPs to bad proxy

    XMLWordPrintable

Details

    • Workaround Exists
    • Hide

      Decrease socket-timeout in mod-cluster-config to minimize impact of down proxies (1-5 seconds):

       <mod-cluster-config advertise-socket="modcluster" proxies="proxy1 proxy2" socket-timeout="5" connector="ajp">
      Show
      Decrease socket-timeout in mod-cluster-config to minimize impact of down proxies (1-5 seconds): <mod-cluster-config advertise-socket= "modcluster" proxies= "proxy1 proxy2" socket-timeout= "5" connector= "ajp" >
    • Hide

      1. Set up mod cluster with multiple proxies. You can further increase the socket-timeout if you wish to exacerbate the issue, but the default socket-timeout of 20 sufficed for me:

      <mod-cluster-config advertise-socket="modcluster" proxies="proxy1 proxy2" socket-timeout="20" connector="ajp">

      2. Add deployments. 5 was enough for concerns in my case.

      3. Leave one or several of the configured mod_cluster proxies in a down/unavailable state. It would need to specifically be down in such a way where JBoss's connection attempts to it are not immediately refused. So you might have to set up some port that is listening but not responding.

      4. Attempt to start JBoss and note slowness and likely the default 300 second blocking timeout will be exceeded.

      Show
      1. Set up mod cluster with multiple proxies. You can further increase the socket-timeout if you wish to exacerbate the issue, but the default socket-timeout of 20 sufficed for me: <mod-cluster-config advertise-socket="modcluster" proxies="proxy1 proxy2" socket-timeout="20" connector="ajp"> 2. Add deployments. 5 was enough for concerns in my case. 3. Leave one or several of the configured mod_cluster proxies in a down/unavailable state. It would need to specifically be down in such a way where JBoss's connection attempts to it are not immediately refused. So you might have to set up some port that is listening but not responding. 4. Attempt to start JBoss and note slowness and likely the default 300 second blocking timeout will be exceeded.

    Description

      There's a timing issue between proxy reset requests and other MCMPs. This can allow for some severe start up delays if there is a configured proxy that is down or unresponsive. Start up is seen to be stalled as it tries to send enable-apps to the bad proxy:

      "ServerService Thread Pool -- 72" #134 prio=5 os_prio=0 tid=0x00007f84581fa9f0 nid=0x6718 runnable [0x00007f83c1561000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
        at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
        at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
        - locked <0x00000000eb0a4700> (a java.io.InputStreamReader)
        at java.io.InputStreamReader.read(InputStreamReader.java:184)
        at java.io.BufferedReader.fill(BufferedReader.java:161)
        at java.io.BufferedReader.readLine(BufferedReader.java:324)
        - locked <0x00000000eb0a4700> (a java.io.InputStreamReader)
        at java.io.BufferedReader.readLine(BufferedReader.java:389)
        at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.sendRequest(DefaultMCMPHandler.java:529)
        at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.sendRequest(DefaultMCMPHandler.java:605)
        - locked <0x00000000fcaac598> (a org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler$Proxy)
        at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.sendRequest(DefaultMCMPHandler.java:429)
        at org.jboss.modcluster.ModClusterService.enable(ModClusterService.java:375)
        at org.jboss.modcluster.ModClusterService.start(ModClusterService.java:361)
        at org.wildfly.mod_cluster.undertow.UndertowEventHandlerAdapter.onDeploymentStart(UndertowEventHandlerAdapter.java:136)
        - locked <0x00000000fcaac7a8> (a org.wildfly.mod_cluster.undertow.UndertowEventHandlerAdapter)
        at org.wildfly.extension.undertow.Host$1.invoke(Host.java:192)
        at org.wildfly.extension.undertow.UndertowService.fireEvent(UndertowService.java:249)
        - locked <0x00000000fcaa4598> (a java.util.Collections$SynchronizedList)
        at org.wildfly.extension.undertow.Host.registerDeployment(Host.java:189)
        at org.wildfly.extension.undertow.deployment.UndertowDeploymentService.startContext(UndertowDeploymentService.java:103)
        at org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:82)

      This can occur because a reset request places the proxy back in an OK state, which then allows the ENABLEs or any MCMP through.

      Attachments

        Issue Links

          incorporates

          Bug - A problem that impairs or prevents the functions of the product. MODCLUSTER-639 proxy reset requests can allow for other MCMPs to bad proxy

          • Critical - To be worked on immediately following BLOCKER issues.
          • Closed
          is incorporated by

          Component Upgrade - A task tracking an update to a bundled component or revision of component upstream of the project. JBEAP-14269 [GSS](7.1.z) Update mod_cluster from 1.3.7.Final to 1.3.9.Final

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed

          Activity

            People

              rhn-engineering-rhusar Radoslav Husar
              rhn-support-aogburn Aaron Ogburn
              Jiří Bílek Jiří Bílek (Inactive)
              Jiří Bílek Jiří Bílek (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: