Uploaded image for project: 'Satellite'
  1. Satellite
  2. SAT-19974

MaxRequestsPerChild from tuning triggers sporadic silent response for clients using HTTP/2

XMLWordPrintable

    • 0
    • False
    • Important
    • None
    • None
    • None
    • None

      Description of problem:
      Satellite uses httpd version vulnerable to https://github.com/apache/httpd/pull/281 bug, where clients using HTTP/2 connections can hit no response from httpd whenever MaxRequestsPerChild is used (and the threshold is just hit).

      That is dangerous due to two reasons:
      1) Investigating the cause is very tricky, as clients wont get any response randomly, and httpd logs do not log anything relevant. Basically enabling httpd debugs is the only option to confirm this.
      2) We do recommend using MaxRequestsPerChild both in performance guide (https://access.redhat.com/documentation/en-us/red_hat_satellite/6.13/html/tuning_performance_of_red_hat_satellite/configuring_project_for_performance_performance-tuning#tuning_apache_httpd_child_processes_performance-tuning), as well as in tuning profiles:

      1. grep maxrequestsperchild /usr/share/foreman-installer/config/foreman.hiera/tuning/sizes/*yaml
        /usr/share/foreman-installer/config/foreman.hiera/tuning/sizes/extra-extra-large.yaml:apache::mod::event::maxrequestsperchild: 4000
        /usr/share/foreman-installer/config/foreman.hiera/tuning/sizes/extra-large.yaml:apache::mod::event::maxrequestsperchild: 4000
        /usr/share/foreman-installer/config/foreman.hiera/tuning/sizes/large.yaml:apache::mod::event::maxrequestsperchild: 4000
        /usr/share/foreman-installer/config/foreman.hiera/tuning/sizes/medium.yaml:apache::mod::event::maxrequestsperchild: 4000
        #

      So the bug can be hit by any customer using HTTP/2 clients (esp. using some automation that very randomly would fail).

      Version-Release number of selected component (if applicable):
      Sat6.13

      • httpd-2.4.37-56.module+el8.8.0+18758+b3a9c8da.6.x86_64

      How reproducible:
      100%

      Steps to Reproduce:
      1. Apply either tuning, or follow the tuning guide directly, to have MaxRequestsPerChild enabled in /etc/httpd/conf.modules.d/event.conf . For the sake of testing, manually decrease the value from 4000 to e.g. 10 or 100 (and restart httpd service)
      2. Run random API requests (or even login page requests) using HTTP/2 protocol, like:

      while true; do
      cnt=0
      while true; do
      cnt=$((cnt+1))
      if [ $((cnt%1000)) -eq 0 ]; then
      echo "running $cnt-th iteration"
      fi
      if [[ $(curl -o /dev/null -s -k --http2 https://localhost/ -w '%

      {size_download}

      ') == 0 ]]; then
      echo "no response received in $cnt-th iteration"
      break
      fi
      done
      sleep 1
      done

      (you can use any URI there, e.g. https://localhost:443/api/v2/status or https://localhost:443/katello/api/v2/organizations/1/ )

      The --http2 option is crucial.

      Actual results:
      2. On average, no response will be received in each MaxRequestsPerChild iteration. Like (for value 100):

      no response received in 127-th iteration
      no response received in 26-th iteration
      no response received in 153-th iteration
      no response received in 82-th iteration
      no response received in 67-th iteration
      no response received in 166-th iteration
      no response received in 86-th iteration
      no response received in 119-th iteration
      no response received in 24-th iteration
      no response received in 191-th iteration
      no response received in 9-th iteration
      no response received in 177-th iteration
      no response received in 47-th iteration
      no response received in 190-th iteration
      no response received in 9-th iteration
      no response received in 144-th iteration

      Expected results:
      The script doesn't print a "no response received" error.

      Additional info:

              ekohlvan@redhat.com Ewoud Kohl van Wijngaarden
              jira-bugzilla-migration RH Bugzilla Integration
              Griffin Sullivan Griffin Sullivan
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: