Critical Description of problem:
Unable to upgrade Red Hat Satellite server 6.15.3 to 6.16. The upgrade failed with below error:
Running satellite-installer : [FAIL]
Failed executing satellite-installer, exit status 1:
2024-09-09 16:26:24 [NOTICE] [root] Loading installer configuration. This will take some time.
2024-09-09 16:26:27 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2024-09-09 16:26:27 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
Package versions are locked. Continuing with unlock.
2024-09-09 16:26:30 [NOTICE] [pre] Performing upgrade of PostgreSQL to 13
2024-09-09 16:26:30 [NOTICE] [pre] Upgrading PostgreSQL packages
2024-09-09 16:27:12 [NOTICE] [pre] Migrating PostgreSQL data
2024-09-09 16:27:30 [ERROR ] [root] runuser -l postgres -c 'postgresql-setup --upgrade' failed! Check the output for error!
--------------------------------------------------------------------------------
Scenario [Migration scripts] failed.
How reproducible:
100%
Is this issue a regression from an earlier version:
NA
Steps to Reproduce:
1. Deploy Satellite 6.15
2. Update the Satellite server to 6.15.z
3. Enable Satellite 6.16 repositories.
4. Initiate upgrade
Actual behavior:
- satellite-maintain upgrade run -y --whitelist="repositories-validate,repositories-setup,non-rh-packages,check-organization-content-access-mode"
Checking for new version of satellite-maintain...
Nothing to update, can't find new version of satellite-maintain.
Running preparation steps required to run the next scenarios
================================================================================
Check whether all services are running: [OK]
--------------------------------------------------------------------------------
Check whether system has any non Red Hat repositories (e.g.: EPEL) enabled:
/ Checking repositories enabled on the system [OK]
--------------------------------------------------------------------------------
Running Checks before upgrading
================================================================================
Check number of fact names in database: [OK]
--------------------------------------------------------------------------------
Clean old Kernel and initramfs files from tftp-boot: [OK]
--------------------------------------------------------------------------------
Check for verifying syntax for ISP DHCP configurations: [OK]
--------------------------------------------------------------------------------
Check whether all services are running: [OK]
--------------------------------------------------------------------------------
Check whether all services are running using the ping call: [OK]
--------------------------------------------------------------------------------
Check for paused tasks: [OK]
--------------------------------------------------------------------------------
Check to verify no empty CA cert requests exist: [OK]
--------------------------------------------------------------------------------
Check whether system is self-registered or not: [OK]
--------------------------------------------------------------------------------
Check to verify if any hotfix installed on system:
/ Checking for presence of hotfix(es). It may take some time to verify.
[OK]
--------------------------------------------------------------------------------
Check if TMOUT environment variable is set: [OK]
--------------------------------------------------------------------------------
Check if any upstream repositories are enabled on system:
| Checking for presence of upstream repositories [OK] -------------------------------------------------------------------------------- Check to make sure root  partition has enough space: [OK]-------------------------------------------------------------------------------- Check to make sure /var/lib/candlepin has enough space: [OK] -------------------------------------------------------------------------------- Make sure server is running on required database version: [OK] -------------------------------------------------------------------------------- Check for roles that have filters with multiple resources attached: [OK] -------------------------------------------------------------------------------- Check for duplicate permissions from database: [OK] -------------------------------------------------------------------------------- Check if system requirements match current tuning profile: [OK] -------------------------------------------------------------------------------- Check whether reports have correct associations: [OK] -------------------------------------------------------------------------------- Check for running tasks: [OK] -------------------------------------------------------------------------------- Check for old tasks in paused/stopped state: [OK] -------------------------------------------------------------------------------- Check for pending tasks which are safe to delete: [OK] -------------------------------------------------------------------------------- Check for tasks in planning state: [OK] -------------------------------------------------------------------------------- Check for running pulpcore tasks: [OK] -------------------------------------------------------------------------------- Check if system has any non Red Hat RPMs installed (e.g.: Fedora): [SKIPPED] -------------------------------------------------------------------------------- Check to validate dnf configuration before upgrade: [OK] -------------------------------------------------------------------------------- Check whether system has any non Red Hat repositories (e.g.: EPEL) enabled:
|
The pre-upgrade checks indicate that the system is ready for upgrade.
It's recommended to perform a backup at this stage.
Confirm to continue with the modification part of the upgrade (assuming yes)
Running preparation steps required to run the next scenarios
================================================================================
Check whether all services are running: [OK]
--------------------------------------------------------------------------------
Running Procedures before migrating
================================================================================
disable active sync plans:
/ Total 0 sync plans are now disabled. [OK]
--------------------------------------------------------------------------------
Add maintenance_mode tables/chain to nftables/iptables: [OK]
--------------------------------------------------------------------------------
Stop cron service:
Stopping the following service(s):
crond
\ All services stopped [OK]
--------------------------------------------------------------------------------
Running Migration scripts
================================================================================
Setup repositories: [SKIPPED]
--------------------------------------------------------------------------------
Switch the given stream modules: [[A[[A [OK]
--------------------------------------------------------------------------------
Enable the given stream modules: [OK]
--------------------------------------------------------------------------------
Download package(s) : [OK]
--------------------------------------------------------------------------------
Stop applicable services:
Stopping the following service(s):
redis, postgresql, pulpcore-api, pulpcore-content, pulpcore-api.socket, pulpcore-content.socket, pulpcore-worker@1.service, pulpcore-worker@2.service, tomcat, dynflow-sidekiq@orchestrator, foreman, httpd, foreman.socket, dynflow-sidekiq@worker-1, dynflow-sidekiq@worker-hosts-queue-1, foreman-proxy
- All services stopped [OK]
--------------------------------------------------------------------------------
Update package(s) : [OK]
--------------------------------------------------------------------------------
Running satellite-installer : [FAIL]
Failed executing satellite-installer, exit status 1:
2024-09-09 16:26:24 [NOTICE] [root] Loading installer configuration. This will take some time.
2024-09-09 16:26:27 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2024-09-09 16:26:27 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
Package versions are locked. Continuing with unlock.
2024-09-09 16:26:30 [NOTICE] [pre] Performing upgrade of PostgreSQL to 13
2024-09-09 16:26:30 [NOTICE] [pre] Upgrading PostgreSQL packages
2024-09-09 16:27:12 [NOTICE] [pre] Migrating PostgreSQL data
2024-09-09 16:27:30 [ERROR ] [root] runuser -l postgres -c 'postgresql-setup --upgrade' failed! Check the output for error!
--------------------------------------------------------------------------------
Scenario [Migration scripts] failed.
The following steps ended up in failing state:
[installer-run]
Resolve the failed steps and rerun the command.
If the situation persists and, you are unclear what to do next,
contact Red Hat Technical Support.
In case the failures are false positives, use
--whitelist="installer-run,repositories-setup"
- runuser -l postgres -c 'postgresql-setup --upgrade'
* Upgrading database.
ERROR: pg_upgrade tool failed
ERROR: Upgrade failed.
* See /var/lib/pgsql/upgrade_postgresql.log for details.
- dnf repolist
Updating Subscription Management repositories.
repo id repo name
Satellite_Engineering_Satellite_6_16_Composes_Satellite_6_16_RHEL8 Satellite 6.16 RHEL8
Satellite_Engineering_Satellite_6_16_Composes_Satellite_Maintenance_6_16_RHEL8 Satellite Maintenance 6.16 RHEL8
rhel-8-for-x86_64-appstream-rpms Red Hat Enterprise Linux 8 for x86_64 - AppStream (RPMs)
rhel-8-for-x86_64-baseos-rpms Red Hat Enterprise Linux 8 for x86_64 - BaseOS (RPMs)
- cat /var/lib/pgsql/upgrade_postgresql.log | tail -100
Performing Consistency Checks
-----------------------------
Checking cluster versions ok
Checking database user is the install user ok
Checking database connection settings ok
Checking for prepared transactions ok
Checking for system-defined composite types in user tables ok
Checking for reg* data types in user tables ok
Checking for contrib/isn with bigint-passing mismatch ok
Creating dump of global objects ok
Creating dump of database schemas
candlepin
foreman
postgres
pulpcore
template1
ok
lc_collate values for database "postgres" do not match: old "en_US.UTF-8", new "en_IN.UTF-8"
Failure, exiting
- dnf repolist
Updating Subscription Management repositories.
repo id repo name
Satellite_Engineering_Satellite_6_16_Composes_Satellite_6_16_RHEL8 Satellite 6.16 RHEL8
Satellite_Engineering_Satellite_6_16_Composes_Satellite_Maintenance_6_16_RHEL8 Satellite Maintenance 6.16 RHEL8
rhel-8-for-x86_64-appstream-rpms Red Hat Enterprise Linux 8 for x86_64 - AppStream (RPMs)
rhel-8-for-x86_64-baseos-rpms Red Hat Enterprise Linux 8 for x86_64 - BaseOS (RPMs)
- localectl
System Locale: LANG=en_IN.UTF-8
VC Keymap: in-eng
X11 Layout: in
X11 Variant: eng
- su - postgres -c "psql -c '\l+'"
psql: error: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
Expected behavior:
- The upgrade should complete successfully without any issue.
Business Impact / Additional info:
