Uploaded image for project: 'Red Hat Advanced Cluster Security'
  1. Red Hat Advanced Cluster Security
  2. ROX-30748

Central migration fails when Central DB has logical replication on

    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • False
    • Rox Sprint 4.9G - Global

      ACSCS failed a migration error for tenant with "logical_replication = 1". The "versions" table has no Pk and thus Postgres cannot identify which row to delete in the replica target.

      There are 8 tables in Central DB without PK:

      | configs
      | delegated_registry_configs
      | installation_infos
      | notification_schedules
      | notifier_enc_configs
      | sensor_upgrade_configs
      | system_infos
      | versions
      
      // Generated by this SQL query:
      SELECT
          t.table_schema,
          t.table_name
      FROM
          information_schema.tables t
      LEFT JOIN
          information_schema.table_constraints tc ON t.table_schema = tc.table_schema
          AND t.table_name = tc.table_name
          AND tc.constraint_type = 'PRIMARY KEY'
      WHERE
          t.table_type = 'BASE TABLE'
          AND t.table_schema NOT IN ('pg_catalog', 'information_schema')
          AND tc.constraint_name IS NULL
      ORDER BY
          t.table_schema,
          t.table_name;

      Full Central logs:

      '/usr/local/share/ca-certificates/00-rds-ca-bundle.crt' -> '/etc/pki/ca-trust/source/anchors/00-rds-ca-bundle.crt''/etc/pki/injected-ca-trust/tls-ca-bundle.pem' -> '/etc/pki/ca-trust/source/anchors/tls-ca-bundle.pem'Migrator: 2025/09/03 14:31:11.973716 log.go:18: Info: Run migrator.run() with version: 4.8.2, DB sequence: 211Migrator: 2025/09/03 14:31:11.974130 log.go:18: Info: conf.Maintenance.ForceRollbackVersion: noneclone/rocksdb: 2025/09/03 14:31:11.976240 db_clone_manager_impl.go:98: Info: Database clones:Migrator: 2025/09/03 14:31:11.976329 log.go:18: Info: connect to gorm: "host=rhacs-d2mrl0t2mn3c73e359rg-db-cluster.cluster-cbdheqtsi6w7.us-east-1.rds.amazonaws.com port=5432 user=rhacs_central dbname=central_active statement_timeout=1200000 client_encoding=UTF8 sslmode=verify-full sslrootcert=/usr/local/share/ca-certificates/00-rds-ca-bundle.crt pool_min_conns=10 pool_max_conns=90"Migrator: 2025/09/03 14:31:12.191989 log.go:18: Info: Migration version from DB = seq_num:211 version:"4.8.1" last_persisted:{seconds:1756216931 nanos:603227000} min_seq_num:186.clone/postgres: 2025/09/03 14:31:12.192281 db_clone_manager_impl.go:221: Info: GetCloneToMigrateMigrator: 2025/09/03 14:31:12.279348 log.go:18: Info: Migration version from DB = seq_num:211 version:"4.8.1" last_persisted:{seconds:1756216931 nanos:603227000} min_seq_num:186.clone/postgres: 2025/09/03 14:31:12.279505 db_clone_manager_impl.go:213: Info: db is of version &{ 4.8.1 211 186 2025-08-26 14:02:11.603227 +0000 UTC}Migrator: 2025/09/03 14:31:12.279532 log.go:18: Info: Clone to Migrate "central_active"Migrator: 2025/09/03 14:31:12.279556 log.go:13: Info: Load databaseMigrator: 2025/09/03 14:31:12.302968 log.go:13: Info: Successfully connected to central database.Migrator: 2025/09/03 14:31:12.358867 log.go:18: Info: Migration version from DB = seq_num:211 version:"4.8.1" last_persisted:{seconds:1756216931 nanos:603227000} min_seq_num:186.Migrator: 2025/09/03 14:31:12.358912 log.go:18: Info: version for "central_active" is &{ 4.8.1 211 186 2025-08-26 14:02:11.603227 +0000 UTC}Migrator: 2025/09/03 14:31:12.358924 log.go:18: Info: In runner.RunMigrator: 2025/09/03 14:31:12.394723 log.go:18: Info: Migration version from DB = seq_num:211 version:"4.8.1" last_persisted:{seconds:1756216931 nanos:603227000} min_seq_num:186.Migrator: 2025/09/03 14:31:12.394756 log.go:18: Info: DB is up to date at version 211. Nothing to do here.panic: retry timer is expired: ERROR: cannot delete from table "versions" because it does not have a replica identity and publishes deletes (SQLSTATE 55000)	failed to write migration version	github.com/stackrox/rox/migrator/version.UpdateVersionPostgres		github.com/stackrox/rox/migrator/version/version.go:70	github.com/stackrox/rox/migrator/runner.updateVersion		github.com/stackrox/rox/migrator/runner/version.go:26	github.com/stackrox/rox/migrator/runner.Run		github.com/stackrox/rox/migrator/runner/runner.go:73	main.upgrade		github.com/stackrox/rox/migrator/upgrade.go:50	main.run		github.com/stackrox/rox/migrator/main.go:88	main.main		github.com/stackrox/rox/migrator/main.go:24	runtime.main		runtime/proc.go:272	runtime.goexit		runtime/asm_amd64.s:1700
      goroutine 1 [running]:github.com/stackrox/rox/pkg/utils/panic.HardPanic(...)	github.com/stackrox/rox/pkg/utils/panic/hard_panic.go:25github.com/stackrox/rox/pkg/utils.CrashOnError(...)	github.com/stackrox/rox/pkg/utils/must.go:20github.com/stackrox/rox/pkg/utils.Must({0xc0038a1ba0?, 0xc0039f2200?, 0x3178ee0?})	github.com/stackrox/rox/pkg/utils/must.go:13 +0x85github.com/stackrox/rox/migrator/version.UpdateVersionPostgres({0x36a82c0?, 0xc003ac73e0?}, {0x36b3ff8?, 0xc001106f70?}, 0x2d263c0?)	github.com/stackrox/rox/migrator/version/version.go:70 +0x88github.com/stackrox/rox/migrator/runner.updateVersion(...)	github.com/stackrox/rox/migrator/runner/version.go:26github.com/stackrox/rox/migrator/runner.Run(0xc003a5c240)	github.com/stackrox/rox/migrator/runner/runner.go:73 +0x2c9main.upgrade({0xc0000e8b80, 0xe})	github.com/stackrox/rox/migrator/upgrade.go:50 +0x2b0main.run()	github.com/stackrox/rox/migrator/main.go:88 +0x267main.main()	github.com/stackrox/rox/migrator/main.go:24 +0x18 

      There are 8 tables without PK:

              rh-ee-dashrews David Shrewsberry
              rh-ee-akurlov Aleksandr Kurlov
              ACS Core Workflows
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated:
                Resolved: