Uploaded image for project: 'mod_cluster'
  1. mod_cluster
  2. MODCLUSTER-427

mod_cluster can break stickiness for the first request on new child processes

XMLWordPrintable

    • Hide

      To exacerbate the issue, run httpd in prefork with a config forcing new processes every request:

      StartServers         1
MaxClients         150
MinSpareServers      1
MaxSpareServers     1 
MaxRequestsPerChild  1

      Set up mod_cluster with "CreateBalancers 1":

      CreateBalancers 1
...
ProxyPass / balancer://mycluster/ stickysession=JSESSIONID|jsessionid nofailover=On
ProxyPassReverse / balancer://mycluster/

      Load up httpd with requests (they should all provide a stickied session) and check for breaks in stickiness. You can confirm the bug occurs also with one node by checking for occurrences of "proxy: byrequests balancer DONE" debug messages for what should be stickied requests.

      Show
      To exacerbate the issue, run httpd in prefork with a config forcing new processes every request: StartServers 1 MaxClients 150 MinSpareServers 1 MaxSpareServers 1 MaxRequestsPerChild 1 Set up mod_cluster with "CreateBalancers 1": CreateBalancers 1 ... ProxyPass / balancer: //mycluster/ stickysession=JSESSIONID|jsessionid nofailover=On ProxyPassReverse / balancer: //mycluster/ Load up httpd with requests (they should all provide a stickied session) and check for breaks in stickiness. You can confirm the bug occurs also with one node by checking for occurrences of "proxy: byrequests balancer DONE" debug messages for what should be stickied requests.
    • Workaround Exists
    • Hide

      -Don't use "CreateBalancers 1"
      -If you do, avoid new child process creation at run time. For prefork, configure MaxSpareServers = StartServers = MaxClients with MaxRequestsPerChild set to 0, for example:

      	MinSpareServers 5
	MaxSpareServers 200
	StartServers 200
	MaxClients 200
	MaxRequestsPerChild 0

      For worker, configure MaxSpareThreads = MaxClients = StartServers*ThreadsPerChild with MaxRequestsPerChild set to 0, for example:

      	ThreadLimit         40
	ServerLimit         5
	StartServers        5
	MinSpareThreads     5
	MaxSpareThreads     200
	MaxClients          200
	ThreadsPerChild     40
	MaxRequestsPerChild 0
      Show
      -Don't use "CreateBalancers 1" -If you do, avoid new child process creation at run time. For prefork, configure MaxSpareServers = StartServers = MaxClients with MaxRequestsPerChild set to 0, for example: MinSpareServers 5 MaxSpareServers 200 StartServers 200 MaxClients 200 MaxRequestsPerChild 0 For worker, configure MaxSpareThreads = MaxClients = StartServers*ThreadsPerChild with MaxRequestsPerChild set to 0, for example: ThreadLimit 40 ServerLimit 5 StartServers 5 MinSpareThreads 5 MaxSpareThreads 200 MaxClients 200 ThreadsPerChild 40 MaxRequestsPerChild 0

      mod_cluster can break stickiness for the first request on new child processes. It looks like this occurs specifically when "CreateBalancers 1" is used. Prefork typically makes this much worse as well.

      My debugging showed that the proxy_balancer would exist, but it would essentially be empty in the new child process. find_session_route/find_route_worker would be called as expected, but the for loop in find_route_worker wasn't even doing anything because balancer->workers->nelts was 0. The balancer would then finally be populated in the new child when the first request hits internal_find_best_byrequests and calls update_workers_node.

              rhn-support-aogburn Aaron Ogburn
              rhn-support-aogburn Aaron Ogburn
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated:
                Resolved: