Description of problem:
--------------------------------

After setting the "install_weak_deps=0" parameter in the "/etc/dnf/dnf.conf" file, when we execute the satellite-installer command for the very first time after installing the Satellite Server, it fails with the following error -

 

[root@sat616rhel95 ~]# satellite-installer --scenario satellite --foreman-initial-organization "ACME" --foreman-initial-admin-username admin --foreman-initial-admin-password redhat
2025-03-09 11:30:09 [NOTICE] [root] Loading installer configuration. This will take some time.
2025-03-09 11:30:20 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2025-03-09 11:30:20 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
2025-03-09 11:32:24 [NOTICE] [configure] Starting system configuration.
2025-03-09 11:33:29 [NOTICE] [configure] 250 configuration steps out of 1523 steps complete.
2025-03-09 11:34:01 [NOTICE] [configure] 500 configuration steps out of 1525 steps complete.
2025-03-09 11:35:07 [NOTICE] [configure] 750 configuration steps out of 1527 steps complete.
2025-03-09 11:35:18 [NOTICE] [configure] 1000 configuration steps out of 1550 steps complete.
2025-03-09 11:36:01 [NOTICE] [configure] 1250 configuration steps out of 1550 steps complete.
2025-03-09 11:39:20 [ERROR ] [configure] Systemd start for httpd failed!
2025-03-09 11:39:20 [ERROR ] [configure] journalctl log for httpd:
2025-03-09 11:39:20 [ERROR ] [configure] Mar 09 11:39:20 sat616rhel95.lab.eng.rdu2.redhat.com systemd[1]: Starting The Apache HTTP Server...
2025-03-09 11:39:20 [ERROR ] [configure] Mar 09 11:39:20 sat616rhel95.lab.eng.rdu2.redhat.com httpd[7849]: httpd: Syntax error on line 40 of /etc/httpd/conf/httpd.conf: Syntax error on line 1 of /etc/httpd/conf.modules.d/http2.load: Cannot load modules/mod_http2.so into server: /etc/httpd/modules/mod_http2.so: cannot open shared object file: No such file or directory
2025-03-09 11:39:20 [ERROR ] [configure] Mar 09 11:39:20 sat616rhel95.lab.eng.rdu2.redhat.com systemd[1]: httpd.service: Main process exited, code=exited, status=1/FAILURE
2025-03-09 11:39:20 [ERROR ] [configure] Mar 09 11:39:20 sat616rhel95.lab.eng.rdu2.redhat.com systemd[1]: httpd.service: Failed with result 'exit-code'.
2025-03-09 11:39:20 [ERROR ] [configure] Mar 09 11:39:20 sat616rhel95.lab.eng.rdu2.redhat.com systemd[1]: Failed to start The Apache HTTP Server.
2025-03-09 11:39:20 [ERROR ] [configure] /Stage[main]/Apache::Service/Service[httpd]/ensure: change from 'stopped' to 'running' failed: Systemd start for httpd failed!
2025-03-09 11:39:20 [ERROR ] [configure] journalctl log for httpd:
2025-03-09 11:39:20 [ERROR ] [configure] Mar 09 11:39:20 sat616rhel95.lab.eng.rdu2.redhat.com systemd[1]: Starting The Apache HTTP Server...
2025-03-09 11:39:20 [ERROR ] [configure] Mar 09 11:39:20 sat616rhel95.lab.eng.rdu2.redhat.com httpd[7849]: httpd: Syntax error on line 40 of /etc/httpd/conf/httpd.conf: Syntax error on line 1 of /etc/httpd/conf.modules.d/http2.load: Cannot load modules/mod_http2.so into server: /etc/httpd/modules/mod_http2.so: cannot open shared object file: No such file or directory
2025-03-09 11:39:20 [ERROR ] [configure] Mar 09 11:39:20 sat616rhel95.lab.eng.rdu2.redhat.com systemd[1]: httpd.service: Main process exited, code=exited, status=1/FAILURE
2025-03-09 11:39:20 [ERROR ] [configure] Mar 09 11:39:20 sat616rhel95.lab.eng.rdu2.redhat.com systemd[1]: httpd.service: Failed with result 'exit-code'.
2025-03-09 11:39:20 [ERROR ] [configure] Mar 09 11:39:20 sat616rhel95.lab.eng.rdu2.redhat.com systemd[1]: Failed to start The Apache HTTP Server.
2025-03-09 11:42:48 [NOTICE] [configure] 1500 configuration steps out of 1550 steps complete.
2025-03-09 11:43:21 [NOTICE] [configure] System configuration has finished.

Error 1: Puppet Service resource 'httpd' failed. Logs:
  /Service[httpd]
    Starting to evaluate the resource (1315 of 1550)
    Skipping restart; service is not running
    Triggered 'refresh' from 1 event
    The container Class[Apache::Service] will propagate my refresh event
    Evaluated in 0.89 seconds
  /Stage[main]/Apache::Service/Service[httpd]/ensure
    change from 'stopped' to 'running' failed: Systemd start for httpd failed!
journalctl log for httpd:
Mar 09 11:39:20 sat616rhel95.lab.eng.rdu2.redhat.com systemd[1]: Starting The Apache HTTP Server...
Mar 09 11:39:20 sat616rhel95.lab.eng.rdu2.redhat.com httpd[7849]: httpd: Syntax error on line 40 of /etc/httpd/conf/httpd.conf: Syntax error on line 1 of /etc/httpd/conf.modules.d/http2.load: Cannot load modules/mod_http2.so into server: /etc/httpd/modules/mod_http2.so: cannot open shared object file: No such file or directory
Mar 09 11:39:20 sat616rhel95.lab.eng.rdu2.redhat.com systemd[1]: httpd.service: Main process exited, code=exited, status=1/FAILURE
Mar 09 11:39:20 sat616rhel95.lab.eng.rdu2.redhat.com systemd[1]: httpd.service: Failed with result 'exit-code'.
Mar 09 11:39:20 sat616rhel95.lab.eng.rdu2.redhat.com systemd[1]: Failed to start The Apache HTTP Server.

1 error was detected during installation.
Please address the errors and re-run the installer to ensure the system is properly configured.
Failing to do so is likely to result in broken functionality.

The full log is at /var/log/foreman-installer/satellite.log
Package versions are being locked.

 

 

How reproducible:
-------------------------

Always.

 

 

Is this issue a regression from an earlier version:
-------------------------------------------------------------------

Not sure as I did not test it.

 

 

Steps to Reproduce:
---------------------------

 

1. Install a fresh RHEL-9.5.

 

2. Add the "install_weak_deps=0" parameter in the "/etc/dnf/dnf.conf" file as follows -

[root@sat616rhel95 ~]# cat /etc/dnf/dnf.conf
[main]
gpgcheck=1
installonly_limit=3
clean_requirements_on_remove=True
best=True
skip_if_unavailable=False
install_weak_deps=0

 

3. Now install the Satellite Server 6.16.

 

4. Run the "satellite-installer command to configure the Satellite Server.

 

 

Actual behavior:
[Describe the issue in detail, including what is happening and where]
------------------------------------------------------------------------------------------------

 

I tried a fresh install of Red Hat Satellite 6.16 on a RHEL-9.5 machine, and it indeed fails.

Here are the results from my freshly installed lab Satellite Server 6.16 on RHEL-9.5 -

[root@sat616rhel95 ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux release 9.5 (Plow)

[root@sat616rhel95 ~]# rpm -qa satellite
satellite-6.16.3-2.el9sat.noarch

[root@sat616rhel95 ~]# rpm -qa mod_http2
[root@sat616rhel95 ~]# 

And here is a full installer log with the final output -

[root@sat616rhel95 ~]# satellite-installer --scenario satellite --foreman-initial-organization "ACME" --foreman-initial-admin-username admin --foreman-initial-admin-password redhat
2025-03-09 11:30:09 [NOTICE] [root] Loading installer configuration. This will take some time.
2025-03-09 11:30:20 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2025-03-09 11:30:20 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
2025-03-09 11:32:24 [NOTICE] [configure] Starting system configuration.
2025-03-09 11:33:29 [NOTICE] [configure] 250 configuration steps out of 1523 steps complete.
2025-03-09 11:34:01 [NOTICE] [configure] 500 configuration steps out of 1525 steps complete.
2025-03-09 11:35:07 [NOTICE] [configure] 750 configuration steps out of 1527 steps complete.
2025-03-09 11:35:18 [NOTICE] [configure] 1000 configuration steps out of 1550 steps complete.
2025-03-09 11:36:01 [NOTICE] [configure] 1250 configuration steps out of 1550 steps complete.
2025-03-09 11:39:20 [ERROR ] [configure] Systemd start for httpd failed!
2025-03-09 11:39:20 [ERROR ] [configure] journalctl log for httpd:
2025-03-09 11:39:20 [ERROR ] [configure] Mar 09 11:39:20 sat616rhel95.lab.eng.rdu2.redhat.com systemd[1]: Starting The Apache HTTP Server...
2025-03-09 11:39:20 [ERROR ] [configure] Mar 09 11:39:20 sat616rhel95.lab.eng.rdu2.redhat.com httpd[7849]: httpd: Syntax error on line 40 of /etc/httpd/conf/httpd.conf: Syntax error on line 1 of /etc/httpd/conf.modules.d/http2.load: Cannot load modules/mod_http2.so into server: /etc/httpd/modules/mod_http2.so: cannot open shared object file: No such file or directory
2025-03-09 11:39:20 [ERROR ] [configure] Mar 09 11:39:20 sat616rhel95.lab.eng.rdu2.redhat.com systemd[1]: httpd.service: Main process exited, code=exited, status=1/FAILURE
2025-03-09 11:39:20 [ERROR ] [configure] Mar 09 11:39:20 sat616rhel95.lab.eng.rdu2.redhat.com systemd[1]: httpd.service: Failed with result 'exit-code'.
2025-03-09 11:39:20 [ERROR ] [configure] Mar 09 11:39:20 sat616rhel95.lab.eng.rdu2.redhat.com systemd[1]: Failed to start The Apache HTTP Server.
2025-03-09 11:39:20 [ERROR ] [configure] /Stage[main]/Apache::Service/Service[httpd]/ensure: change from 'stopped' to 'running' failed: Systemd start for httpd failed!
2025-03-09 11:39:20 [ERROR ] [configure] journalctl log for httpd:
2025-03-09 11:39:20 [ERROR ] [configure] Mar 09 11:39:20 sat616rhel95.lab.eng.rdu2.redhat.com systemd[1]: Starting The Apache HTTP Server...
2025-03-09 11:39:20 [ERROR ] [configure] Mar 09 11:39:20 sat616rhel95.lab.eng.rdu2.redhat.com httpd[7849]: httpd: Syntax error on line 40 of /etc/httpd/conf/httpd.conf: Syntax error on line 1 of /etc/httpd/conf.modules.d/http2.load: Cannot load modules/mod_http2.so into server: /etc/httpd/modules/mod_http2.so: cannot open shared object file: No such file or directory
2025-03-09 11:39:20 [ERROR ] [configure] Mar 09 11:39:20 sat616rhel95.lab.eng.rdu2.redhat.com systemd[1]: httpd.service: Main process exited, code=exited, status=1/FAILURE
2025-03-09 11:39:20 [ERROR ] [configure] Mar 09 11:39:20 sat616rhel95.lab.eng.rdu2.redhat.com systemd[1]: httpd.service: Failed with result 'exit-code'.
2025-03-09 11:39:20 [ERROR ] [configure] Mar 09 11:39:20 sat616rhel95.lab.eng.rdu2.redhat.com systemd[1]: Failed to start The Apache HTTP Server.
2025-03-09 11:42:48 [NOTICE] [configure] 1500 configuration steps out of 1550 steps complete.
2025-03-09 11:43:21 [NOTICE] [configure] System configuration has finished.

Error 1: Puppet Service resource 'httpd' failed. Logs:
  /Service[httpd]
    Starting to evaluate the resource (1315 of 1550)
    Skipping restart; service is not running
    Triggered 'refresh' from 1 event
    The container Class[Apache::Service] will propagate my refresh event
    Evaluated in 0.89 seconds
  /Stage[main]/Apache::Service/Service[httpd]/ensure
    change from 'stopped' to 'running' failed: Systemd start for httpd failed!
journalctl log for httpd:
Mar 09 11:39:20 sat616rhel95.lab.eng.rdu2.redhat.com systemd[1]: Starting The Apache HTTP Server...
Mar 09 11:39:20 sat616rhel95.lab.eng.rdu2.redhat.com httpd[7849]: httpd: Syntax error on line 40 of /etc/httpd/conf/httpd.conf: Syntax error on line 1 of /etc/httpd/conf.modules.d/http2.load: Cannot load modules/mod_http2.so into server: /etc/httpd/modules/mod_http2.so: cannot open shared object file: No such file or directory
Mar 09 11:39:20 sat616rhel95.lab.eng.rdu2.redhat.com systemd[1]: httpd.service: Main process exited, code=exited, status=1/FAILURE
Mar 09 11:39:20 sat616rhel95.lab.eng.rdu2.redhat.com systemd[1]: httpd.service: Failed with result 'exit-code'.
Mar 09 11:39:20 sat616rhel95.lab.eng.rdu2.redhat.com systemd[1]: Failed to start The Apache HTTP Server.

1 error was detected during installation.
Please address the errors and re-run the installer to ensure the system is properly configured.
Failing to do so is likely to result in broken functionality.

The full log is at /var/log/foreman-installer/satellite.log
Package versions are being locked.

 

The installer completed unsuccessfully with signal 6 as can be seen from the logs below -

 

[root@sat616rhel95 ~]# vim /var/log/foreman-installer/satellite.log
-bash: vim: command not found
[root@sat616rhel95 ~]# 
[root@sat616rhel95 ~]# tail -10 /var/log/foreman-installer/satellite.log
2025-03-09 11:43:26 [DEBUG ] [root] --------------------------------------------------------------------------------
2025-03-09 11:43:26 [DEBUG ] [root] 
2025-03-09 11:43:26 [DEBUG ] [post] Hook /usr/share/foreman-installer/hooks/post/99-version_locking.rb returned nil
2025-03-09 11:43:26 [INFO  ] [post] All hooks in group post finished
2025-03-09 11:43:26 [DEBUG ] [pre_exit] Hook /usr/share/foreman-installer/hooks/pre_exit/20-certs_regenerate.rb returned nil
2025-03-09 11:43:26 [DEBUG ] [root] Exit with status code: 6 (signal was 6)
2025-03-09 11:43:26 [DEBUG ] [root] Cleaning /tmp/kafo_installation20250309-4250-f4moko
2025-03-09 11:43:26 [DEBUG ] [root] Cleaning /tmp/kafo_installation20250309-4250-aam1s2
2025-03-09 11:43:26 [DEBUG ] [root] Cleaning /tmp/default_values.yaml
2025-03-09 11:43:26 [DEBUG ] [root] Installer finished in 786.911305628 seconds

 

 

Expected behavior:
[Describe what should be happening instead]
---------------------------------------------------------------

 

Satellite Server should get configured correctly as expected.

 

 

Business Impact / Additional info:
----------------------------------------------

 

Satellite Server 6.16's installation failure.

 

 

Reproducer Details -
--------------------------

 

Satellite Server 6.16 IP   -  10.8.29.5
SSH Login Command      -  ajambhul@10.8.29.5
Credentials                       -  ajambhul/ajambhul

 

 

Additional Information -
--------------------------------

 

After manually installing the "mod_http2" rpm it works correctly as expected.

Check out the logs below -

[root@sat616rhel95 ~]# satellite-maintain packages install mod_http2
Running install packages in unlocked session
================================================================================
Confirm installer run is allowed: 

WARNING: This script runs satellite-installer after the dnf execution 
to ensure the Satellite is in a consistent state.
As a result some of your services may be restarted. 

Do you want to proceed?, [y(yes), q(quit)] y
                                                                      [OK]      
--------------------------------------------------------------------------------
Unlock packages:                                                      [OK]
--------------------------------------------------------------------------------
Install packages: Updating Subscription Management repositories.
Last metadata expiration check: 0:56:43 ago on Sun 09 Mar 2025 11:20:22 AM EDT.
Dependencies resolved.
================================================================================
 Package    Arch    Version             Repository                         Size
================================================================================
Installing:
 mod_http2  x86_64  2.0.26-2.el9_4.1    rhel-9-for-x86_64-appstream-rpms  167 k

Transaction Summary
================================================================================
Install  1 Package

Total download size: 167 k
Installed size: 442 k
Is this ok [y/N]: y
Downloading Packages:
mod_http2-2.0.26-2.el9_4.1.x86_64.rpm           702 kB/s | 167 kB     00:00    
--------------------------------------------------------------------------------
Total                                           693 kB/s | 167 kB     00:00     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                        1/1 
  Installing       : mod_http2-2.0.26-2.el9_4.1.x86_64                      1/1 
  Running scriptlet: mod_http2-2.0.26-2.el9_4.1.x86_64                      1/1 
  Verifying        : mod_http2-2.0.26-2.el9_4.1.x86_64                      1/1 
Installed products updated.

Installed:
  mod_http2-2.0.26-2.el9_4.1.x86_64                                             

Complete!
                                                    [OK]
--------------------------------------------------------------------------------
Running satellite-installer : 2025-03-09 12:17:17 [NOTICE] [root] Loading installer configuration. This will take some time.
2025-03-09 12:17:23 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2025-03-09 12:17:23 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
2025-03-09 12:17:36 [NOTICE] [configure] Starting system configuration.
2025-03-09 12:17:50 [NOTICE] [configure] 250 configuration steps out of 1523 steps complete.
2025-03-09 12:17:52 [NOTICE] [configure] 500 configuration steps out of 1525 steps complete.
2025-03-09 12:17:56 [NOTICE] [configure] 750 configuration steps out of 1527 steps complete.
2025-03-09 12:17:57 [NOTICE] [configure] 1000 configuration steps out of 1536 steps complete.
2025-03-09 12:17:58 [NOTICE] [configure] 1250 configuration steps out of 1536 steps complete.
2025-03-09 12:22:13 [NOTICE] [configure] 1500 configuration steps out of 1536 steps complete.
2025-03-09 12:22:24 [NOTICE] [configure] System configuration has finished.
  Success!
  * Satellite is running at https://sat616rhel95.lab.eng.rdu2.redhat.com
      Initial credentials are admin / redhat

  * To install an additional Capsule on separate machine continue by running:

      capsule-certs-generate --foreman-proxy-fqdn "$CAPSULE" --certs-tar "/root/$CAPSULE-certs.tar"
  * Capsule is running at https://sat616rhel95.lab.eng.rdu2.redhat.com:9090

The full log is at /var/log/foreman-installer/satellite.log
Package versions are being locked.
                                        [OK]
--------------------------------------------------------------------------------
Check status of version locking of packages: 
  Automatic locking of package versions is enabled in installer.
  Packages are locked.                                                [OK]
--------------------------------------------------------------------------------

[root@sat616rhel95 ~]# 

 

 

Customer's Notes -
-------------------------

Please refer case's problem description for additional information/analysis from the customer.