Overview

The showcase-runtime namespace deployment consistently fails in the pull-ci-redhat-developer-rhdh-release-1.7-e2e-ocp-operator-nightly job due to database migration file corruption.

Error Details

Error: The migration directory is corrupt, the following files are missing:
20250514000000_refresh_state_references_big_increments.js
20250707164600_user_created_at.js

AggregateError: Backend startup failed

Pod: backstage-rhdh-* (showcase-runtime namespace)
Container: backstage-backend

Complete error location: Pod logs can be found in job artifacts under showcase-runtime/pod_logs/backstage-rhdh-<hash>_backstage-backend-previous.log

Root Cause

The shared external PostgreSQL RDS database contains migration records from a newer RHDH version, but the RHDH 1.7 container being tested does not include these migration files. When Knex.js migration framework performs its integrity check, it detects the missing files and fails the backend startup.

Missing migration files in RHDH 1.7:

• 20250514000000_refresh_state_references_big_increments.js
• 20250707164600_user_created_at.js

Environment Details

• Job: pull-ci-redhat-developer-rhdh-release-1.7-e2e-ocp-operator-nightly
• Branch: release-1.7
• Helm Chart Version: RHDH 1.7.x
• Database: External PostgreSQL RDS (shared across test runs)
• Namespace: showcase-runtime
• Deployment Type: OpenShift Operator

Impact

• Blocks all showcase-runtime configuration change tests from running
• Test sequence: showcase (runs) → showcase-rbac (runs) → showcase-runtime (FAILS)
• This is an infrastructure/environment issue, not a code defect

Related Issues

Steps to Reproduce

1. Run Prow job: pull-ci-redhat-developer-rhdh-release-1.7-e2e-ocp-operator-nightly
2. Wait for showcase and showcase-rbac deployments to complete
3. Observe showcase-runtime deployment enter CrashLoopBackOff
4. Check pod logs: oc logs <pod-name> -c backstage-backend -n showcase-runtime

Example Job Runs

• PR #3802 Run 1
• PR #3802 Run 3

Workaround

Reset/clean the shared RDS database to remove migration records from newer RHDH versions, or use ephemeral databases that match the RHDH version being tested.

Recommended Fix

1. Ensure test environments use dedicated databases per RHDH version
2. Implement database cleanup between test runs
3. Add migration validation checks before test execution
4. Consider using ephemeral PostgreSQL instances instead of shared RDS

Additional Information

Version Information

• RHDH Version: 1.7.x
• Helm Chart: oci://quay.io/rhdh/chart (version 1.7.x)
• Deployment Method: OpenShift Operator

Key Files

• .ibm/pipelines/jobs/ocp-operator.sh:37-52 - run_operator_runtime_config_change_tests()
• .ibm/pipelines/resources/postgres-db/postgres-cred.yaml - RDS credentials
• .ibm/pipelines/resources/postgres-db/rds-app-config.yaml - Database configuration