Uploaded image for project: 'JBoss Core Services'
  1. JBoss Core Services
  2. JBCS-197

BalancerMember directives don't work and cause SegFaults

    XMLWordPrintable

Details

    • 32
    • +
    • Hide

      See description

      Show
      See description

    Description

      There has been an ongoing discussion about interoperability between BalancerMember and ProxyPass directives and mod_cluster. This is a follow up on MODCLUSTER-391 and especially MODCLUSTER-356.

      TL;DR

      • BalancerMember directives don't work as expected (at all)
      • it is possible to use it to cause SegFault in httpd
      • If these directives are supposed to work, then I have a wrong configuration or it is a bug to be fixed
      • If they are not supposed to work in conjunction with mod_cluster, then I should stop trying to test these and remove all ever-failing scenarios from the test suite

      Configuration and goal

      • two web apps, clusterbench.war and tses.war, both deployed on each of two tomcats
      • one web app is in excluded contexts (it is tses.war)
      • the other one ([^clusterbench]) is registered with mod_cluster balancer
      • main server: *:2080
      • mod_cluster VirtualHost: *:8747
      • proxyPass BalancerMember VirtualHost *:2081
      • I want to access clusterbench.war via *:8747 and *:2080 (works ), and tses.war via *:2081 (fails )
      • see proxy_test.conf for BalancerMember configuration (taken from httpd 2.2.26 test run, you must edit Location access)
      • see mod_cluster.conf for mod_cluster configuration (taken from httpd 2.2.26 test run, as above)

      Test

      Intermittent and stable SegFaults

      httpd 2.2.26 / mod_cluster 1.2.13.Final (EWS 2.1.1)

      With the aforementioned setup, it is possible to cause SegFault roughly in 50% of requests to curl http://192.168.122.172:2081/clusterbench/requestinfo on httpd 2.2.26 mod_cluster 1.2.13.Final, the rest passes fine and the web app is served.

      Offending line: mod_proxy_cluster.c:3843
      Trace:

      #0  proxy_cluster_pre_request (worker=<optimized out>, balancer=<optimized out>, r=0x5555558be3e0, conf=0x5555558767d8, url=0x7fffffffdd40) at mod_proxy_cluster.c:3843
#1  0x00007ffff0cfe3d6 in proxy_run_pre_request (worker=worker@entry=0x7fffffffdd38, balancer=balancer@entry=0x7fffffffdd30, r=r@entry=0x5555558be3e0, 
    conf=conf@entry=0x5555558767d8, url=url@entry=0x7fffffffdd40) at /builddir/build/BUILD/httpd-EWS_2.1.1.CR1/modules/proxy/mod_proxy.c:2428
#2  0x00007ffff0d01ef2 in ap_proxy_pre_request (worker=worker@entry=0x7fffffffdd38, balancer=balancer@entry=0x7fffffffdd30, r=r@entry=0x5555558be3e0, 
    conf=conf@entry=0x5555558767d8, url=url@entry=0x7fffffffdd40) at /builddir/build/BUILD/httpd-EWS_2.1.1.CR1/modules/proxy/proxy_util.c:1512
#3  0x00007ffff0cfeabb in proxy_handler (r=0x5555558be3e0) at /builddir/build/BUILD/httpd-EWS_2.1.1.CR1/modules/proxy/mod_proxy.c:952
#4  0x00005555555805e0 in ap_run_handler (r=0x5555558be3e0) at /builddir/build/BUILD/httpd-EWS_2.1.1.CR1/server/config.c:157
#5  0x00005555555809a9 in ap_invoke_handler (r=r@entry=0x5555558be3e0) at /builddir/build/BUILD/httpd-EWS_2.1.1.CR1/server/config.c:376
#6  0x000055555558dc58 in ap_process_request (r=r@entry=0x5555558be3e0) at /builddir/build/BUILD/httpd-EWS_2.1.1.CR1/modules/http/http_request.c:282
#7  0x000055555558aff8 in ap_process_http_connection (c=0x5555558ae2f0) at /builddir/build/BUILD/httpd-EWS_2.1.1.CR1/modules/http/http_core.c:190
#8  0x0000555555587010 in ap_run_process_connection (c=0x5555558ae2f0) at /builddir/build/BUILD/httpd-EWS_2.1.1.CR1/server/connection.c:43
#9  0x00005555555873b0 in ap_process_connection (c=c@entry=0x5555558ae2f0, csd=<optimized out>) at /builddir/build/BUILD/httpd-EWS_2.1.1.CR1/server/connection.c:190
#10 0x0000555555592b5b in child_main (child_num_arg=child_num_arg@entry=0) at /builddir/build/BUILD/httpd-EWS_2.1.1.CR1/server/mpm/prefork/prefork.c:667
#11 0x0000555555592fae in make_child (s=0x5555557bf880, slot=0) at /builddir/build/BUILD/httpd-EWS_2.1.1.CR1/server/mpm/prefork/prefork.c:712
#12 0x0000555555593b6e in ap_mpm_run (_pconf=_pconf@entry=0x5555557ba158, plog=<optimized out>, s=s@entry=0x5555557bf880)
    at /builddir/build/BUILD/httpd-EWS_2.1.1.CR1/server/mpm/prefork/prefork.c:988
#13 0x000055555556b50e in main (argc=8, argv=0x7fffffffe268) at /builddir/build/BUILD/httpd-EWS_2.1.1.CR1/server/main.c:753

      httpd 2.4.23 / mod_cluster 1.3.3.Final (JBCS 2.4.23)

      With the aforementioned setup, it is always possible to SegFault httpd by accessing tses.war on BalancerMember ProxyPass VirtualHos: curl http://192.168.122.172:2081/tses.

      Offending line: mod_proxy_cluster.c:2230
      Trace:

      #0  0x00007fffe61a598f in internal_find_best_byrequests (balancer=0x55555593ad38, conf=0x555555918dd8, r=0x5555559a6630, domain=0x0, failoverdomain=0, 
    vhost_table=0x5555559a5c98, context_table=0x5555559a5e00, node_table=0x5555559a6088) at mod_proxy_cluster.c:2230
#1  0x00007fffe61a90c8 in find_best_worker (balancer=0x55555593ad38, conf=0x555555918dd8, r=0x5555559a6630, domain=0x0, failoverdomain=0, vhost_table=0x5555559a5c98, 
    context_table=0x5555559a5e00, node_table=0x5555559a6088, recurse=1) at mod_proxy_cluster.c:3457
#2  0x00007fffe61a9f4d in proxy_cluster_pre_request (worker=0x7fffffffdb68, balancer=0x7fffffffdb60, r=0x5555559a6630, conf=0x555555918dd8, url=0x7fffffffdb70)
    at mod_proxy_cluster.c:3825
#3  0x00007fffec2fd9a6 in proxy_run_pre_request (worker=worker@entry=0x7fffffffdb68, balancer=balancer@entry=0x7fffffffdb60, r=r@entry=0x5555559a6630, 
    conf=conf@entry=0x555555918dd8, url=url@entry=0x7fffffffdb70) at mod_proxy.c:2853
#4  0x00007fffec302652 in ap_proxy_pre_request (worker=worker@entry=0x7fffffffdb68, balancer=balancer@entry=0x7fffffffdb60, r=r@entry=0x5555559a6630, 
    conf=conf@entry=0x555555918dd8, url=url@entry=0x7fffffffdb70) at proxy_util.c:1956
#5  0x00007fffec2fe1dc in proxy_handler (r=0x5555559a6630) at mod_proxy.c:1108
#6  0x00005555555aeff0 in ap_run_handler (r=r@entry=0x5555559a6630) at config.c:170
#7  0x00005555555af539 in ap_invoke_handler (r=r@entry=0x5555559a6630) at config.c:434
#8  0x00005555555c5b2a in ap_process_async_request (r=0x5555559a6630) at http_request.c:410
#9  0x00005555555c5e04 in ap_process_request (r=r@entry=0x5555559a6630) at http_request.c:445
#10 0x00005555555c1ded in ap_process_http_sync_connection (c=0x555555950050) at http_core.c:210
#11 ap_process_http_connection (c=0x555555950050) at http_core.c:251
#12 0x00005555555b9470 in ap_run_process_connection (c=c@entry=0x555555950050) at connection.c:42
#13 0x00005555555b99c8 in ap_process_connection (c=c@entry=0x555555950050, csd=<optimized out>) at connection.c:226
#14 0x00007fffec513a30 in child_main (child_num_arg=child_num_arg@entry=0, child_bucket=child_bucket@entry=0) at prefork.c:723
#15 0x00007fffec513c70 in make_child (s=0x55555582d400, slot=slot@entry=0, bucket=bucket@entry=0) at prefork.c:767
#16 0x00007fffec51521d in prefork_run (_pconf=<optimized out>, plog=0x5555558313a8, s=0x55555582d400) at prefork.c:979
#17 0x0000555555592aae in ap_run_mpm (pconf=pconf@entry=0x555555804188, plog=0x5555558313a8, s=0x55555582d400) at mpm_common.c:94
#18 0x000055555558bb18 in main (argc=8, argv=0x7fffffffe1a8) at main.c:783

      About the test

      This test has always been failing in one way or another: not serving URL (HTTP 404), returning All workers in Error state (HTTP 503). SegFault has been slipping under the radar for some time, because the test ended up on assert earlier in the scenario - on the first HTTP 503.
      We should clearly document which BalancerMember integration is supported and which is not. Furthermore, we must not SegFault even if user tries to do something weird, we must log an error message instead.

      Attachments

        1. clusterbench.war
          392 kB
        2. mod_cluster.conf
          1.0 kB
        3. proxy_test.conf
          0.7 kB
        4. tses.war
          2 kB

        Issue Links

          causes

          Bug - A problem that impairs or prevents the functions of the product. JBCS-854 mod_cluster + mod_proxy_wstunnel doesn't work - all workers are in an error state

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed
          clones

          Bug - A problem that impairs or prevents the functions of the product. MODCLUSTER-543 BalancerMember directives don't work and cause SegFaults

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Resolved
          incorporates

          Bug - A problem that impairs or prevents the functions of the product. JBCS-854 mod_cluster + mod_proxy_wstunnel doesn't work - all workers are in an error state

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed
          is incorporated by

          Bug - A problem that impairs or prevents the functions of the product. JBCS-763 Upgrade mod_cluster (native) to 1.3.11.Final-redhat

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed

          Activity

            People

              gzaronik@redhat.com George Zaronikas
              mbabacek1@redhat.com Michal Karm
              Paul Lodge Paul Lodge
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: