Uploaded image for project: 'mod_cluster'
  1. mod_cluster
  2. MODCLUSTER-427

mod_cluster can break stickiness for the first request on new child processes

    XMLWordPrintable

    Details

    • Steps to Reproduce:
      Hide

      To exacerbate the issue, run httpd in prefork with a config forcing new processes every request:

      StartServers         1
      MaxClients         150
      MinSpareServers      1
      MaxSpareServers     1 
      MaxRequestsPerChild  1
      

      Set up mod_cluster with "CreateBalancers 1":

      CreateBalancers 1
      ...
      ProxyPass / balancer://mycluster/ stickysession=JSESSIONID|jsessionid nofailover=On
      ProxyPassReverse / balancer://mycluster/
      

      Load up httpd with requests (they should all provide a stickied session) and check for breaks in stickiness. You can confirm the bug occurs also with one node by checking for occurrences of "proxy: byrequests balancer DONE" debug messages for what should be stickied requests.

      Show
      To exacerbate the issue, run httpd in prefork with a config forcing new processes every request: StartServers 1 MaxClients 150 MinSpareServers 1 MaxSpareServers 1 MaxRequestsPerChild 1 Set up mod_cluster with "CreateBalancers 1": CreateBalancers 1 ... ProxyPass / balancer: //mycluster/ stickysession=JSESSIONID|jsessionid nofailover=On ProxyPassReverse / balancer: //mycluster/ Load up httpd with requests (they should all provide a stickied session) and check for breaks in stickiness. You can confirm the bug occurs also with one node by checking for occurrences of "proxy: byrequests balancer DONE" debug messages for what should be stickied requests.
    • Workaround:
      Workaround Exists
    • Workaround Description:
      Hide

      -Don't use "CreateBalancers 1"
      -If you do, avoid new child process creation at run time. For prefork, configure MaxSpareServers = StartServers = MaxClients with MaxRequestsPerChild set to 0, for example:

      	MinSpareServers 5
      	MaxSpareServers 200
      	StartServers 200
      	MaxClients 200
      	MaxRequestsPerChild 0
      

      For worker, configure MaxSpareThreads = MaxClients = StartServers*ThreadsPerChild with MaxRequestsPerChild set to 0, for example:

      	ThreadLimit         40
      	ServerLimit         5
      	StartServers        5
      	MinSpareThreads     5
      	MaxSpareThreads     200
      	MaxClients          200
      	ThreadsPerChild     40
      	MaxRequestsPerChild 0
      
      Show
      -Don't use "CreateBalancers 1" -If you do, avoid new child process creation at run time. For prefork, configure MaxSpareServers = StartServers = MaxClients with MaxRequestsPerChild set to 0, for example: MinSpareServers 5 MaxSpareServers 200 StartServers 200 MaxClients 200 MaxRequestsPerChild 0 For worker, configure MaxSpareThreads = MaxClients = StartServers*ThreadsPerChild with MaxRequestsPerChild set to 0, for example: ThreadLimit 40 ServerLimit 5 StartServers 5 MinSpareThreads 5 MaxSpareThreads 200 MaxClients 200 ThreadsPerChild 40 MaxRequestsPerChild 0

      Description

      mod_cluster can break stickiness for the first request on new child processes. It looks like this occurs specifically when "CreateBalancers 1" is used. Prefork typically makes this much worse as well.

      My debugging showed that the proxy_balancer would exist, but it would essentially be empty in the new child process. find_session_route/find_route_worker would be called as expected, but the for loop in find_route_worker wasn't even doing anything because balancer->workers->nelts was 0. The balancer would then finally be populated in the new child when the first request hits internal_find_best_byrequests and calls update_workers_node.

        Attachments

          Activity

            People

            Assignee:
            aogburn Aaron Ogburn
            Reporter:
            aogburn Aaron Ogburn
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: