Created attachment 2026028 [details]
satellite.log file attached for further reference

Description of problem:
----------------------

While executing the "satellite-installer --scenario satellite" command after the installation of Red Hat Satellite 6.16 on a FIPS-enabled RHEL-9.3 system, foreman-proxy service fails to start resulting in the following error message -

[root@sat616testrhel9fips ~]# satellite-installer --scenario satellite --foreman-initial-admin-username admin --foreman-initial-admin-password admin_password
2024-04-10 03:25:08 [NOTICE] [root] Loading installer configuration. This will take some time.
2024-04-10 03:25:13 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2024-04-10 03:25:13 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
2024-04-10 03:26:59 [NOTICE] [configure] Starting system configuration.
2024-04-10 03:27:53 [NOTICE] [configure] 250 configuration steps out of 1608 steps complete.
2024-04-10 03:28:57 [NOTICE] [configure] 500 configuration steps out of 1609 steps complete.
2024-04-10 03:29:18 [NOTICE] [configure] 750 configuration steps out of 1611 steps complete.
2024-04-10 03:30:55 [NOTICE] [configure] 1000 configuration steps out of 1637 steps complete.
2024-04-10 03:31:27 [NOTICE] [configure] 1250 configuration steps out of 1639 steps complete.
2024-04-10 03:38:52 [NOTICE] [configure] 1500 configuration steps out of 1639 steps complete.
2024-04-10 03:42:54 [ERROR ] [configure] Systemd start for foreman-proxy failed!
2024-04-10 03:42:54 [ERROR ] [configure] journalctl log for foreman-proxy:
2024-04-10 03:42:54 [ERROR ] [configure] Apr 10 03:42:51 sat616testrhel9fips.lab.eng.rdu2.redhat.com systemd[1]: Starting Foreman Proxy...
2024-04-10 03:42:54 [ERROR ] [configure] Apr 10 03:42:53 sat616testrhel9fips.lab.eng.rdu2.redhat.com smart-proxy[28033]: Errors detected on startup, see log for details. Exiting: Could not parse PKey
2024-04-10 03:42:54 [ERROR ] [configure] Apr 10 03:42:53 sat616testrhel9fips.lab.eng.rdu2.redhat.com systemd[1]: foreman-proxy.service: Main process exited, code=exited, status=1/FAILURE
2024-04-10 03:42:54 [ERROR ] [configure] Apr 10 03:42:53 sat616testrhel9fips.lab.eng.rdu2.redhat.com systemd[1]: foreman-proxy.service: Failed with result 'exit-code'.
2024-04-10 03:42:54 [ERROR ] [configure] Apr 10 03:42:53 sat616testrhel9fips.lab.eng.rdu2.redhat.com systemd[1]: Failed to start Foreman Proxy.
2024-04-10 03:42:54 [ERROR ] [configure] Apr 10 03:42:53 sat616testrhel9fips.lab.eng.rdu2.redhat.com systemd[1]: foreman-proxy.service: Consumed 2.456s CPU time.
2024-04-10 03:42:54 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Service/Service[foreman-proxy]/ensure: change from 'stopped' to 'running' failed: Systemd start for foreman-proxy failed!
2024-04-10 03:42:54 [ERROR ] [configure] journalctl log for foreman-proxy:
2024-04-10 03:42:54 [ERROR ] [configure] Apr 10 03:42:51 sat616testrhel9fips.lab.eng.rdu2.redhat.com systemd[1]: Starting Foreman Proxy...
2024-04-10 03:42:54 [ERROR ] [configure] Apr 10 03:42:53 sat616testrhel9fips.lab.eng.rdu2.redhat.com smart-proxy[28033]: Errors detected on startup, see log for details. Exiting: Could not parse PKey
2024-04-10 03:42:54 [ERROR ] [configure] Apr 10 03:42:53 sat616testrhel9fips.lab.eng.rdu2.redhat.com systemd[1]: foreman-proxy.service: Main process exited, code=exited, status=1/FAILURE
2024-04-10 03:42:54 [ERROR ] [configure] Apr 10 03:42:53 sat616testrhel9fips.lab.eng.rdu2.redhat.com systemd[1]: foreman-proxy.service: Failed with result 'exit-code'.
2024-04-10 03:42:54 [ERROR ] [configure] Apr 10 03:42:53 sat616testrhel9fips.lab.eng.rdu2.redhat.com systemd[1]: Failed to start Foreman Proxy.
2024-04-10 03:42:54 [ERROR ] [configure] Apr 10 03:42:53 sat616testrhel9fips.lab.eng.rdu2.redhat.com systemd[1]: foreman-proxy.service: Consumed 2.456s CPU time.
2024-04-10 03:43:04 [NOTICE] [configure] System configuration has finished.

Error 1: Puppet Service resource 'foreman-proxy' failed. Logs:
/Service[foreman-proxy]
Starting to evaluate the resource (1621 of 1639)
Skipping restart; service is not running
Triggered 'refresh' from 5 events
The container Class[Foreman_proxy::Service] will propagate my refresh event
Evaluated in 4.84 seconds
/Stage[main]/Foreman_proxy::Service/Service[foreman-proxy]/ensure
change from 'stopped' to 'running' failed: Systemd start for foreman-proxy failed!
journalctl log for foreman-proxy:
Apr 10 03:42:51 sat616testrhel9fips.lab.eng.rdu2.redhat.com systemd[1]: Starting Foreman Proxy...
Apr 10 03:42:53 sat616testrhel9fips.lab.eng.rdu2.redhat.com smart-proxy[28033]: Errors detected on startup, see log for details. Exiting: Could not parse PKey
Apr 10 03:42:53 sat616testrhel9fips.lab.eng.rdu2.redhat.com systemd[1]: foreman-proxy.service: Main process exited, code=exited, status=1/FAILURE
Apr 10 03:42:53 sat616testrhel9fips.lab.eng.rdu2.redhat.com systemd[1]: foreman-proxy.service: Failed with result 'exit-code'.
Apr 10 03:42:53 sat616testrhel9fips.lab.eng.rdu2.redhat.com systemd[1]: Failed to start Foreman Proxy.
Apr 10 03:42:53 sat616testrhel9fips.lab.eng.rdu2.redhat.com systemd[1]: foreman-proxy.service: Consumed 2.456s CPU time.

1 error was detected during installation.
Please address the errors and re-run the installer to ensure the system is properly configured.
Failing to do so is likely to result in broken functionality.

The full log is at /var/log/foreman-installer/satellite.log
Package versions are being locked.
[root@sat616testrhel9fips ~]#

Version-Release number of selected component (if applicable):
------------------------------------------------------------

1. Red Hat Satellite 6.16 -

[root@sat616testrhel9fips ~]# rpm -qa satellite
satellite-6.16.0-0.2.stream.el9sat.noarch

2. Red Hat Enterprise Linux 9 -

[root@sat616testrhel9fips ~]# cat /etc/redhat-release
Red Hat Enterprise Linux release 9.3 (Plow)

3. FIPS-enabled while the installation of RHEL-9.3 by modifying the kernel command line -

[root@sat616testrhel9fips ~]# fips-mode-setup --check
FIPS mode is enabled.

How reproducible:
----------------

Always

Steps to Reproduce:
------------------

1. Install RHEL-9 and enable FIPS mode while the installation process is going on. I have followed the steps to enable FIPS mode as outlined in the following official documentation -

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html-single/security_hardening/index#assembly_installing-the-system-in-fips-mode_security-hardening

2. Install Red Hat Satellite 6.16 by following the steps as outlined in the following internal documentation -

https://ohsnap.sat.engineering.redhat.com/streams/stream/releases/stream/snaps/51.0/installation

3. Execute the following command as outlined in the above documentation to further configure Red Hat Satellite 6.16 -

1. satellite-installer --scenario satellite --foreman-initial-admin-username admin --foreman-initial-admin-password admin_password

Actual results:
--------------

Configuring Satellite Server 6.16 fails to start the foreman-proxy service.

Expected results:
----------------

Configuring Satellite Server 6.16 should successfully start foreman-proxy as well as the entire Red Hat Satellite services stack successfully.

Additional info:
---------------

Following errors can be observed in the "/var/log/foreman-installer/satellite.log" file -

2024-04-10 03:42:54 [ERROR ] [configure] Systemd start for foreman-proxy failed!
2024-04-10 03:42:54 [ERROR ] [configure] journalctl log for foreman-proxy:
2024-04-10 03:42:54 [ERROR ] [configure] Apr 10 03:42:51 sat616testrhel9fips.lab.eng.rdu2.redhat.com systemd[1]: Starting Foreman Proxy...
2024-04-10 03:42:54 [ERROR ] [configure] Apr 10 03:42:53 sat616testrhel9fips.lab.eng.rdu2.redhat.com smart-proxy[28033]: Errors detected on startup, see log for details. Exiting: Could not parse PKey
2024-04-10 03:42:54 [ERROR ] [configure] Apr 10 03:42:53 sat616testrhel9fips.lab.eng.rdu2.redhat.com systemd[1]: foreman-proxy.service: Main process exited, code=exited, status=1/FAILURE
2024-04-10 03:42:54 [ERROR ] [configure] Apr 10 03:42:53 sat616testrhel9fips.lab.eng.rdu2.redhat.com systemd[1]: foreman-proxy.service: Failed with result 'exit-code'.
2024-04-10 03:42:54 [ERROR ] [configure] Apr 10 03:42:53 sat616testrhel9fips.lab.eng.rdu2.redhat.com systemd[1]: Failed to start Foreman Proxy.
2024-04-10 03:42:54 [ERROR ] [configure] Apr 10 03:42:53 sat616testrhel9fips.lab.eng.rdu2.redhat.com systemd[1]: foreman-proxy.service: Consumed 2.456s CPU time.
2024-04-10 03:42:54 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Service/Service[foreman-proxy]/ensure: change from 'stopped' to 'running' failed: Systemd start for foreman-proxy failed!
2024-04-10 03:42:54 [ERROR ] [configure] journalctl log for foreman-proxy:
2024-04-10 03:42:54 [ERROR ] [configure] Apr 10 03:42:51 sat616testrhel9fips.lab.eng.rdu2.redhat.com systemd[1]: Starting Foreman Proxy...
2024-04-10 03:42:54 [ERROR ] [configure] Apr 10 03:42:53 sat616testrhel9fips.lab.eng.rdu2.redhat.com smart-proxy[28033]: Errors detected on startup, see log for details. Exiting: Could not parse PKey
2024-04-10 03:42:54 [ERROR ] [configure] Apr 10 03:42:53 sat616testrhel9fips.lab.eng.rdu2.redhat.com systemd[1]: foreman-proxy.service: Main process exited, code=exited, status=1/FAILURE
2024-04-10 03:42:54 [ERROR ] [configure] Apr 10 03:42:53 sat616testrhel9fips.lab.eng.rdu2.redhat.com systemd[1]: foreman-proxy.service: Failed with result 'exit-code'.
2024-04-10 03:42:54 [ERROR ] [configure] Apr 10 03:42:53 sat616testrhel9fips.lab.eng.rdu2.redhat.com systemd[1]: Failed to start Foreman Proxy.
2024-04-10 03:42:54 [ERROR ] [configure] Apr 10 03:42:53 sat616testrhel9fips.lab.eng.rdu2.redhat.com systemd[1]: foreman-proxy.service: Consumed 2.456s CPU time.
2024-04-10 03:42:54 [DEBUG ] [configure] Executing: '/bin/systemctl is-active – foreman-proxy'
2024-04-10 03:42:54 [DEBUG ] [configure] /Service[foreman-proxy]: Skipping restart; service is not running
.....

satellite-installer finished with the status code: 6 as follows -

024-04-10 03:43:11 [DEBUG ] [post] Hook /usr/share/foreman-installer/hooks/post/99-version_locking.rb returned nil
2024-04-10 03:43:11 [INFO ] [post] All hooks in group post finished
2024-04-10 03:43:11 [DEBUG ] [pre_exit] Hook /usr/share/foreman-installer/hooks/pre_exit/20-certs_regenerate.rb returned nil
2024-04-10 03:43:11 [DEBUG ] [root] Exit with status code: 6 (signal was 6)
2024-04-10 03:43:11 [DEBUG ] [root] Cleaning /tmp/kafo_installation20240410-20342-u8i92f
2024-04-10 03:43:11 [DEBUG ] [root] Cleaning /tmp/kafo_installation20240410-20342-3cgzz8
2024-04-10 03:43:11 [DEBUG ] [root] Cleaning /tmp/default_values.yaml
2024-04-10 03:43:11 [DEBUG ] [root] Installer finished in 1078.051543351 seconds

Additionally, I am also attaching "/var/log/foreman-installer/satellite.log" file to the BZ for further evaluation.