-
Bug
-
Resolution: Done
-
Major
-
6.13.0
Description of problem:
Satellite uses httpd version vulnerable to https://github.com/apache/httpd/pull/281 bug, where clients using HTTP/2 connections can hit no response from httpd whenever MaxRequestsPerChild is used (and the threshold is just hit).
That is dangerous due to two reasons:
1) Investigating the cause is very tricky, as clients wont get any response randomly, and httpd logs do not log anything relevant. Basically enabling httpd debugs is the only option to confirm this.
2) We do recommend using MaxRequestsPerChild both in performance guide (https://access.redhat.com/documentation/en-us/red_hat_satellite/6.13/html/tuning_performance_of_red_hat_satellite/configuring_project_for_performance_performance-tuning#tuning_apache_httpd_child_processes_performance-tuning), as well as in tuning profiles:
- grep maxrequestsperchild /usr/share/foreman-installer/config/foreman.hiera/tuning/sizes/*yaml
/usr/share/foreman-installer/config/foreman.hiera/tuning/sizes/extra-extra-large.yaml:apache::mod::event::maxrequestsperchild: 4000
/usr/share/foreman-installer/config/foreman.hiera/tuning/sizes/extra-large.yaml:apache::mod::event::maxrequestsperchild: 4000
/usr/share/foreman-installer/config/foreman.hiera/tuning/sizes/large.yaml:apache::mod::event::maxrequestsperchild: 4000
/usr/share/foreman-installer/config/foreman.hiera/tuning/sizes/medium.yaml:apache::mod::event::maxrequestsperchild: 4000
#
So the bug can be hit by any customer using HTTP/2 clients (esp. using some automation that very randomly would fail).
Version-Release number of selected component (if applicable):
Sat6.13
- httpd-2.4.37-56.module+el8.8.0+18758+b3a9c8da.6.x86_64
How reproducible:
100%
Steps to Reproduce:
1. Apply either tuning, or follow the tuning guide directly, to have MaxRequestsPerChild enabled in /etc/httpd/conf.modules.d/event.conf . For the sake of testing, manually decrease the value from 4000 to e.g. 10 or 100 (and restart httpd service)
2. Run random API requests (or even login page requests) using HTTP/2 protocol, like:
while true; do
cnt=0
while true; do
cnt=$((cnt+1))
if [ $((cnt%1000)) -eq 0 ]; then
echo "running $cnt-th iteration"
fi
if [[ $(curl -o /dev/null -s -k --http2 https://localhost/ -w '%
') == 0 ]]; then
echo "no response received in $cnt-th iteration"
break
fi
done
sleep 1
done
(you can use any URI there, e.g. https://localhost:443/api/v2/status or https://localhost:443/katello/api/v2/organizations/1/ )
The --http2 option is crucial.
Actual results:
2. On average, no response will be received in each MaxRequestsPerChild iteration. Like (for value 100):
no response received in 127-th iteration
no response received in 26-th iteration
no response received in 153-th iteration
no response received in 82-th iteration
no response received in 67-th iteration
no response received in 166-th iteration
no response received in 86-th iteration
no response received in 119-th iteration
no response received in 24-th iteration
no response received in 191-th iteration
no response received in 9-th iteration
no response received in 177-th iteration
no response received in 47-th iteration
no response received in 190-th iteration
no response received in 9-th iteration
no response received in 144-th iteration
Expected results:
The script doesn't print a "no response received" error.
Additional info: