-
Bug
-
Resolution: Done
-
Critical
-
httpd 2.4.23 ER1
-
32
-
-
-
-
-
-
+
-
There has been an ongoing discussion about interoperability between BalancerMember and ProxyPass directives and mod_cluster. This is a follow up on MODCLUSTER-391 and especially MODCLUSTER-356.
TL;DR
- BalancerMember directives don't work as expected (at all)
- it is possible to use it to cause SegFault in httpd
- If these directives are supposed to work, then I have a wrong configuration or it is a bug to be fixed
- If they are not supposed to work in conjunction with mod_cluster, then I should stop trying to test these and remove all ever-failing scenarios from the test suite
Configuration and goal
- two web apps, clusterbench.war
and tses.war, both deployed on each of two tomcats - one web app is in excluded contexts (it is tses.war)
- the other one ([^clusterbench]) is registered with mod_cluster balancer
- main server: *:2080
- mod_cluster VirtualHost: *:8747
- proxyPass BalancerMember VirtualHost *:2081
- I want to access clusterbench.war via *:8747 and *:2080 (works
), and tses.war via *:2081 (fails 
) - see proxy_test.conf for BalancerMember configuration (taken from httpd 2.2.26 test run, you must edit Location access)
- see mod_cluster.conf for mod_cluster configuration (taken from httpd 2.2.26 test run, as above)
Test
- check, that only clusterbench.war is registered and everything is cool: mod_cluster-manager console
- clusterbench.war on mod_cluster VirtualHost works: curl http://192.168.122.172:8747/clusterbench/requestinfo
- clusterbench.war on main server also works: curl http://192.168.122.172:2080/clusterbench/requestinfo (it works due to
MODCLUSTER-430) - httpd 2.2.26 / mod_cluster 1.2.13.Final:
- tses.war on BalancerMember ProxyPass VirtualHost fails: curl http://192.168.122.172:2081/tses with:
mod_proxy_cluster.c(2374): proxy: byrequests balancer FAILED proxy: CLUSTER: (balancer://xqacluster). All workers are in error state
and it doesn't matter whether I configure the same balancer (qacluster) for both mod_cluster and additional BalancerMemebr directives or if I have two balancers (this case).
- clusterbench.war on BalancerMember ProxyPass VirtualHost sometimes works and sometimes causes SegFault curl http://192.168.122.172:2081/clusterbench/requestinfo (see below)
- httpd 2.4.23 / mod_cluster 1.3.3.Final:
- tses.war on BalancerMember ProxyPass VirtualHost fails with curl http://192.168.122.172:2081/tses SegFault, always (see below)
- clusterbench.war on BalancerMember ProxyPass VirtualHost works curl http://192.168.122.172:2081/clusterbench/requestinfo
Intermittent and stable SegFaults
httpd 2.2.26 / mod_cluster 1.2.13.Final (EWS 2.1.1)
With the aforementioned setup, it is possible to cause SegFault roughly in 50% of requests to curl http://192.168.122.172:2081/clusterbench/requestinfo on httpd 2.2.26 mod_cluster 1.2.13.Final, the rest passes fine and the web app is served.
Offending line: mod_proxy_cluster.c:3843
Trace:
#0 proxy_cluster_pre_request (worker=<optimized out>, balancer=<optimized out>, r=0x5555558be3e0, conf=0x5555558767d8, url=0x7fffffffdd40) at mod_proxy_cluster.c:3843
#1 0x00007ffff0cfe3d6 in proxy_run_pre_request (worker=worker@entry=0x7fffffffdd38, balancer=balancer@entry=0x7fffffffdd30, r=r@entry=0x5555558be3e0,
conf=conf@entry=0x5555558767d8, url=url@entry=0x7fffffffdd40) at /builddir/build/BUILD/httpd-EWS_2.1.1.CR1/modules/proxy/mod_proxy.c:2428
#2 0x00007ffff0d01ef2 in ap_proxy_pre_request (worker=worker@entry=0x7fffffffdd38, balancer=balancer@entry=0x7fffffffdd30, r=r@entry=0x5555558be3e0,
conf=conf@entry=0x5555558767d8, url=url@entry=0x7fffffffdd40) at /builddir/build/BUILD/httpd-EWS_2.1.1.CR1/modules/proxy/proxy_util.c:1512
#3 0x00007ffff0cfeabb in proxy_handler (r=0x5555558be3e0) at /builddir/build/BUILD/httpd-EWS_2.1.1.CR1/modules/proxy/mod_proxy.c:952
#4 0x00005555555805e0 in ap_run_handler (r=0x5555558be3e0) at /builddir/build/BUILD/httpd-EWS_2.1.1.CR1/server/config.c:157
#5 0x00005555555809a9 in ap_invoke_handler (r=r@entry=0x5555558be3e0) at /builddir/build/BUILD/httpd-EWS_2.1.1.CR1/server/config.c:376
#6 0x000055555558dc58 in ap_process_request (r=r@entry=0x5555558be3e0) at /builddir/build/BUILD/httpd-EWS_2.1.1.CR1/modules/http/http_request.c:282
#7 0x000055555558aff8 in ap_process_http_connection (c=0x5555558ae2f0) at /builddir/build/BUILD/httpd-EWS_2.1.1.CR1/modules/http/http_core.c:190
#8 0x0000555555587010 in ap_run_process_connection (c=0x5555558ae2f0) at /builddir/build/BUILD/httpd-EWS_2.1.1.CR1/server/connection.c:43
#9 0x00005555555873b0 in ap_process_connection (c=c@entry=0x5555558ae2f0, csd=<optimized out>) at /builddir/build/BUILD/httpd-EWS_2.1.1.CR1/server/connection.c:190
#10 0x0000555555592b5b in child_main (child_num_arg=child_num_arg@entry=0) at /builddir/build/BUILD/httpd-EWS_2.1.1.CR1/server/mpm/prefork/prefork.c:667
#11 0x0000555555592fae in make_child (s=0x5555557bf880, slot=0) at /builddir/build/BUILD/httpd-EWS_2.1.1.CR1/server/mpm/prefork/prefork.c:712
#12 0x0000555555593b6e in ap_mpm_run (_pconf=_pconf@entry=0x5555557ba158, plog=<optimized out>, s=s@entry=0x5555557bf880)
at /builddir/build/BUILD/httpd-EWS_2.1.1.CR1/server/mpm/prefork/prefork.c:988
#13 0x000055555556b50e in main (argc=8, argv=0x7fffffffe268) at /builddir/build/BUILD/httpd-EWS_2.1.1.CR1/server/main.c:753
httpd 2.4.23 / mod_cluster 1.3.3.Final (JBCS 2.4.23)
With the aforementioned setup, it is always possible to SegFault httpd by accessing tses.war on BalancerMember ProxyPass VirtualHos: curl http://192.168.122.172:2081/tses.
Offending line: mod_proxy_cluster.c:2230
Trace:
#0 0x00007fffe61a598f in internal_find_best_byrequests (balancer=0x55555593ad38, conf=0x555555918dd8, r=0x5555559a6630, domain=0x0, failoverdomain=0,
vhost_table=0x5555559a5c98, context_table=0x5555559a5e00, node_table=0x5555559a6088) at mod_proxy_cluster.c:2230
#1 0x00007fffe61a90c8 in find_best_worker (balancer=0x55555593ad38, conf=0x555555918dd8, r=0x5555559a6630, domain=0x0, failoverdomain=0, vhost_table=0x5555559a5c98,
context_table=0x5555559a5e00, node_table=0x5555559a6088, recurse=1) at mod_proxy_cluster.c:3457
#2 0x00007fffe61a9f4d in proxy_cluster_pre_request (worker=0x7fffffffdb68, balancer=0x7fffffffdb60, r=0x5555559a6630, conf=0x555555918dd8, url=0x7fffffffdb70)
at mod_proxy_cluster.c:3825
#3 0x00007fffec2fd9a6 in proxy_run_pre_request (worker=worker@entry=0x7fffffffdb68, balancer=balancer@entry=0x7fffffffdb60, r=r@entry=0x5555559a6630,
conf=conf@entry=0x555555918dd8, url=url@entry=0x7fffffffdb70) at mod_proxy.c:2853
#4 0x00007fffec302652 in ap_proxy_pre_request (worker=worker@entry=0x7fffffffdb68, balancer=balancer@entry=0x7fffffffdb60, r=r@entry=0x5555559a6630,
conf=conf@entry=0x555555918dd8, url=url@entry=0x7fffffffdb70) at proxy_util.c:1956
#5 0x00007fffec2fe1dc in proxy_handler (r=0x5555559a6630) at mod_proxy.c:1108
#6 0x00005555555aeff0 in ap_run_handler (r=r@entry=0x5555559a6630) at config.c:170
#7 0x00005555555af539 in ap_invoke_handler (r=r@entry=0x5555559a6630) at config.c:434
#8 0x00005555555c5b2a in ap_process_async_request (r=0x5555559a6630) at http_request.c:410
#9 0x00005555555c5e04 in ap_process_request (r=r@entry=0x5555559a6630) at http_request.c:445
#10 0x00005555555c1ded in ap_process_http_sync_connection (c=0x555555950050) at http_core.c:210
#11 ap_process_http_connection (c=0x555555950050) at http_core.c:251
#12 0x00005555555b9470 in ap_run_process_connection (c=c@entry=0x555555950050) at connection.c:42
#13 0x00005555555b99c8 in ap_process_connection (c=c@entry=0x555555950050, csd=<optimized out>) at connection.c:226
#14 0x00007fffec513a30 in child_main (child_num_arg=child_num_arg@entry=0, child_bucket=child_bucket@entry=0) at prefork.c:723
#15 0x00007fffec513c70 in make_child (s=0x55555582d400, slot=slot@entry=0, bucket=bucket@entry=0) at prefork.c:767
#16 0x00007fffec51521d in prefork_run (_pconf=<optimized out>, plog=0x5555558313a8, s=0x55555582d400) at prefork.c:979
#17 0x0000555555592aae in ap_run_mpm (pconf=pconf@entry=0x555555804188, plog=0x5555558313a8, s=0x55555582d400) at mpm_common.c:94
#18 0x000055555558bb18 in main (argc=8, argv=0x7fffffffe1a8) at main.c:783
About the test
This test has always been failing in one way or another: not serving URL (HTTP 404), returning All workers in Error state (HTTP 503). SegFault has been slipping under the radar for some time, because the test ended up on assert earlier in the scenario - on the first HTTP 503.
We should clearly document which BalancerMember integration is supported and which is not. Furthermore, we must not SegFault even if user tries to do something weird, we must log an error message instead.
- causes
-
-
- Closed
-
- clones
-
-
- Resolved
-
- incorporates
-
-
- Closed
-
- is incorporated by
-
-
- Closed
-
