USER PROBLEM
After customer migrated from RHACS 4.4.1  to 4.8.0 ,Central migration stuck in a loop when trying to delete entries from the table compliance_operator_rule_v2

• Checked pods running in stackrox namespace, which indicates the central and config-controller pods are NOT running

```
$ oc get pods -n stackrox
NAME READY STATUS RESTARTS AGE
admission-control-67946b667c-7d4jn 1/1 Running 0 13m
admission-control-67946b667c-fsl4l 1/1 Running 0 2h
admission-control-67946b667c-h8blf 1/1 Running 0 13m
central-75f9b646f-cvpr4 0/1 Running 1 7m
central-db-6848fb5694-tm9lz 1/1 Running 0 11m
collector-2dxpb 3/3 Running 0 3d
collector-4kkmr 3/3 Running 5 3d
collector-5h7zr 3/3 Running 0 3d
collector-6lf5f 3/3 Running 0 3d
collector-dxjgg 3/3 Running 0 3d
collector-jcpfk 3/3 Running 0 3d
collector-k6xps 3/3 Running 0 3d
collector-km7wm 3/3 Running 0 3d
collector-kmps7 3/3 Running 0 3d
collector-qbgfw 3/3 Running 0 3d
collector-rjgh9 3/3 Running 0 3d
collector-rqgbg 3/3 Running 0 3d
collector-vlbd2 3/3 Running 0 3d
collector-w544h 3/3 Running 0 3d
collector-wmm57 3/3 Running 0 3d
collector-wmtck 3/3 Running 0 3d
collector-zwf9z 3/3 Running 0 3d
config-controller-8496c587c5-b8mgq 0/1 Running 9 2h
scanner-6b57c67c6d-2kchx 1/1 Running 0 1h
scanner-6b57c67c6d-n99ps 1/1 Running 0 13m
scanner-db-74d99f7694-5m877 1/1 Running 0 13m
sensor-7bb88ff7b8-9jcvn 1/1 Running 0 1h

• Checked central logs
  ```
  2025-08-25T16:42:32.239577567+08:00 '/usr/local/share/ca-certificates/ca.crt' -> '/etc/pki/ca-trust/source/anchors/ca.crt'
  2025-08-25T16:42:32.243783540+08:00 '/etc/pki/injected-ca-trust/tls-ca-bundle.pem' -> '/etc/pki/ca-trust/source/anchors/tls-ca-bundle.pem'
  2025-08-25T16:42:33.598050926+08:00 Migrator: 2025/08/25 08:42:33.597830 log.go:18: Info: Run migrator.run() with version: 4.8.0, DB sequence: 211
  2025-08-25T16:42:33.598312247+08:00 Migrator: 2025/08/25 08:42:33.598268 log.go:18: Info: conf.Maintenance.ForceRollbackVersion: none
  2025-08-25T16:42:33.599040470+08:00 Migrator: 2025/08/25 08:42:33.598987 log.go:18: Info: checking if the database "central_active" exists
  2025-08-25T16:42:33.611420512+08:00 clone/rocksdb: 2025/08/25 08:42:33.611304 db_clone_manager_impl.go:98: Info: Database clones:
  2025-08-25T16:42:33.611471647+08:00 Migrator: 2025/08/25 08:42:33.611432 log.go:18: Info: connect to gorm: "host=central-db.stackrox.svc port=5432 user=postgres sslmode=verify-full sslrootcert=/run/secrets/stackrox.io/certs/ca.pem statement_timeout=1.2e+06 pool_min_conns=10 pool_max_conns=90 client_encoding=UTF8"
  2025-08-25T16:42:33.666882051+08:00 Migrator: 2025/08/25 08:42:33.666780 log.go:18: Info: Migration version from DB = seq_num:200.
  2025-08-25T16:42:33.667132483+08:00 clone/postgres: 2025/08/25 08:42:33.667036 db_clone_manager_impl.go:94: Info: db is of version &{ 200 0 1970-01-01 00:00:00 +0000 UTC}
  2025-08-25T16:42:33.687920000+08:00 clone/postgres: 2025/08/25 08:42:33.687825 db_clone_manager_impl.go:167: Info: safeRemove -> central_temp
  2025-08-25T16:42:33.704826592+08:00 NOTICE: database "central_temp" does not exist, skipping
  2025-08-25T16:42:33.705629825+08:00 clone/postgres: 2025/08/25 08:42:33.705545 db_clone_manager_impl.go:158: Info: Postgres Database clones:
  2025-08-25T16:42:33.705670303+08:00 clone/postgres: 2025/08/25 08:42:33.705619 db_clone_manager_impl.go:160: Info: central_active -> &{ 200 0 1970-01-01 00:00:00 +0000 UTC}
  2025-08-25T16:42:33.705699208+08:00 clone/postgres: 2025/08/25 08:42:33.705662 db_clone_manager_impl.go:221: Info: GetCloneToMigrate
  2025-08-25T16:42:33.705722868+08:00 clone/postgres: 2025/08/25 08:42:33.705697 db_clone_manager_impl.go:167: Info: safeRemove -> central_previous

• 2025-08-25T16:42:33.721348969+08:00 NOTICE: database "central_previous" does not exist, skipping
  2025-08-25T16:42:33.766014206+08:00 clone/postgres: 2025/08/25 08:42:33.765912 db_clone_manager_impl.go:477: Info: Central has space to create backup for rollback: true, required: 19268012601, available: 268803108864 with 0.200000 margin
  2025-08-25T16:42:33.766014206+08:00 Migrator: 2025/08/25 08:42:33.765971 log.go:18: Info: Clone to Migrate "central_active"
  2025-08-25T16:42:33.766029827+08:00 Migrator: 2025/08/25 08:42:33.765995 log.go:18: Info: Load database = "central_active"
  2025-08-25T16:42:33.785612472+08:00 Migrator: 2025/08/25 08:42:33.785528 log.go:13: Info: Successfully connected to central database.
  2025-08-25T16:42:33.833709357+08:00 Migrator: 2025/08/25 08:42:33.833608 log.go:18: Info: Migration version from DB = seq_num:200.
  2025-08-25T16:42:33.833742232+08:00 Migrator: 2025/08/25 08:42:33.833691 log.go:18: Info: version for "central_active" is &{ 200 0 1970-01-01 00:00:00 +0000 UTC}
  2025-08-25T16:42:33.833742232+08:00 Migrator: 2025/08/25 08:42:33.833714 log.go:18: Info: In runner.Run
  2025-08-25T16:42:33.861430377+08:00 Migrator: 2025/08/25 08:42:33.861344 log.go:18: Info: Migration version from DB = seq_num:200.
  2025-08-25T16:42:33.861430377+08:00 Migrator: 2025/08/25 08:42:33.861400 log.go:18: Info: Found DB at version 200, which is less than what we expect (211). Running migrations...
  2025-08-25T16:42:34.165252334+08:00 migrations/m_200_to_m_201_compliance_v2_for_4_5: 2025/08/25 08:42:34.165101 migration_impl.go:103: Info: Converted 814 profile records
  2025-08-25T16:42:34.242540654+08:00 pkg/search/postgres: 2025/08/25 08:42:34.242410 common.go:1147: Error: Query issue: delete from compliance_operator_rule_v2 where compliance_operator_rule_v2.Id = ANY($1::text[]): conn busy
  2025-08-25T16:42:39.243391062+08:00 pkg/search/postgres: 2025/08/25 08:42:39.243278 common.go:1147: Error: Query issue: delete from compliance_operator_rule_v2 where compliance_operator_rule_v2.Id = ANY($1::text[]): conn busy
  2025-08-25T16:42:44.247628515+08:00 pkg/search/postgres: 2025/08/25 08:42:44.247525 common.go:1147: Error: Query issue: delete from compliance_operator_rule_v2 where compliance_operator_rule_v2.Id = ANY($1::text[]): conn busy
  2025-08-25T16:42:49.246812424+08:00 pkg/search/postgres: 2025/08/25 08:42:49.246718 common.go:1147: Error: Query issue: delete from compliance_operator_rule_v2 where compliance_operator_rule_v2.Id = ANY($1::text[]): conn busy
  2025-08-25T16:42:54.247024521+08:00 pkg/search/postgres: 2025/08/25 08:42:54.246923 common.go:1147: Error: Query issue: delete from compliance_operator_rule_v2 where compliance_operator_rule_v2.Id = ANY($1::text[]): conn busy
  2025-08-25T16:42:59.246876870+08:00 pkg/search/postgres: 2025/08/25 08:42:59.246743 common.go:1147: Error: Query issue: delete from compliance_operator_rule_v2 where compliance_operator_rule_v2.Id = ANY($1::text[]): conn busy
  2025-08-25T16:43:04.246855270+08:00 pkg/search/postgres: 2025/08/25 08:43:04.246727 common.go:1147: Error: Query issue: delete from compliance_operator_rule_v2 where compliance_operator_rule_v2.Id = ANY($1::text[]): conn busy
  2025-08-25T16:43:09.243842163+08:00 pkg/search/postgres: 2025/08/25 08:43:09.243735 common.go:1147: Error: Query issue: delete from compliance_operator_rule_v2 where compliance_operator_rule_v2.Id = ANY($1::text[]): conn busy
  2025-08-25T16:43:14.246832142+08:00 pkg/search/postgres: 2025/08/25 08:43:14.246720 common.go:1147: Error: Query issue: delete from compliance_operator_rule_v2 where compliance_operator_rule_v2.Id = ANY($1::text[]): conn busy
  2025-08-25T16:43:19.247097085+08:00 pkg/search/postgres: 2025/08/25 08:43:19.246940 common.go:1147: Error: Query issue: delete from compliance_operator_rule_v2 where compliance_operator_rule_v2.Id = ANY($1::text[]): conn busy
  2025-08-25T16:43:24.247257113+08:00 pkg/search/postgres: 2025/08/25 08:43:24.247168 common.go:1147: Error: Query issue: delete from compliance_operator_rule_v2 where compliance_operator_rule_v2.Id = ANY($1::text[]): conn busy
  2025-08-25T16:43:29.247575781+08:00 pkg/search/postgres: 2025/08/25 08:43:29.247447 common.go:1147: Error: Query issue: delete from compliance_operator_rule_v2 where compliance_operator_rule_v2.Id = ANY($1::text[]): conn busy
  2025-08-25T16:43:34.242821507+08:00 pkg/search/postgres: 2025/08/25 08:43:34.242694 common.go:1147: Error: Query issue: delete from compliance_operator_rule_v2 where compliance_operator_rule_v2.Id = ANY($1::text[]): conn busy
  2025-08-25T16:43:39.247151725+08:00 pkg/search/postgres: 2025/08/25 08:43:39.247033 common.go:1147: Error: Query issue: delete from compliance_operator_rule_v2 where compliance_operator_rule_v2.Id = ANY($1::text[]): conn busy
  2025-08-25T16:43:44.246868814+08:00 pkg/search/postgres: 2025/08/25 08:43:44.246775 common.go:1147: Error: Query issue: delete from compliance_operator_rule_v2 where compliance_operator_rule_v2.Id = ANY($1::text[]): conn busy
  2025-08-25T16:43:49.243290523+08:00 pkg/search/postgres: 2025/08/25 08:43:49.243207 common.go:1147: Error: Query issue: delete from compliance_operator_rule_v2 where compliance_operator_rule_v2.Id = ANY($1::text[]): conn busy
  2025-08-25T16:43:54.247346173+08:00 pkg/search/postgres: 2025/08/25 08:43:54.247260 common.go:1147: Error: Query issue: delete from compliance_operator_rule_v2 where compliance_operator_rule_v2.Id = ANY($1::text[]): conn busy
  2025-08-25T16:43:59.247253524+08:00 pkg/search/postgres: 2025/08/25 08:43:59.247158 common.go:1147: Error: Query issue: delete from compliance_operator_rule_v2 where compliance_operator_rule_v2.Id = ANY($1::text[]): conn busy
  2025-08-25T16:44:04.242812789+08:00 pkg/search/postgres: 2025/08/25 08:44:04.242714 common.go:1147: Error: Query issue: delete from compliance_operator_rule_v2 where compliance_operator_rule_v2.Id = ANY($1::text[]): conn busy
  2025-08-25T16:44:09.247161449+08:00 pkg/search/postgres: 2025/08/25 08:44:09.247030 common.go:1147: Error: Query issue: delete from compliance_operator_rule_v2 where compliance_operator_rule_v2.Id = ANY($1::text[]): conn busy
  2025-08-25T16:44:14.247472303+08:00 pkg/search/postgres: 2025/08/25 08:44:14.247355 common.go:1147: Error: Query issue: delete from compliance_operator_rule_v2 where compliance_operator_rule_v2.Id = ANY($1::text[]): conn busy
  2025-08-25T16:44:19.242865047+08:00 pkg/search/postgres: 2025/08/25 08:44:19.242760 common.go:1147: Error: Query issue: delete from compliance_operator_rule_v2 where compliance_operator_rule_v2.Id = ANY($1::text[]): conn busy
  2025-08-25T16:44:24.247073811+08:00 pkg/search/postgres: 2025/08/25 08:44:24.246982 common.go:1147: Error: Query issue: delete from compliance_operator_rule_v2 where compliance_operator_rule_v2.Id = ANY($1::text[]): conn busy
  2025-08-25T16:44:29.247355402+08:00 pkg/search/postgres: 2025/08/25 08:44:29.247269 common.go:1147: Error: Query issue: delete from compliance_operator_rule_v2 where compliance_operator_rule_v2.Id = ANY($1::text[]): conn busy
  2025-08-25T16:44:34.243085039+08:00 pkg/search/postgres: 2025/08/25 08:44:34.242721 common.go:1147: Error: Query issue: delete from compliance_operator_rule_v2 where compliance_operator_rule_v2.Id = ANY($1::text[]): conn busy
  2025-08-25T16:44:39.242826473+08:00 pkg/search/postgres: 2025/08/25 08:44:39.242729 common.go:1147: Error: Query issue: delete from compliance_operator_rule_v2 where compliance_operator_rule_v2.Id = ANY($1::text[]): conn busy
  2025-08-25T16:44:44.247323451+08:00 pkg/search/postgres: 2025/08/25 08:44:44.247235 common.go:1147: Error: Query issue: delete from compliance_operator_rule_v2 where compliance_operator_rule_v2.Id = ANY($1::text[]): conn busy
  2025-08-25T16:44:49.247598322+08:00 pkg/search/postgres: 2025/08/25 08:44:49.247505 common.go:1147: Error: Query issue: delete from compliance_operator_rule_v2 where compliance_operator_rule_v2.Id = ANY($1::text[]): conn busy
  2025-08-25T16:44:54.246896767+08:00 pkg/search/postgres: 2025/08/25 08:44:54.246804 common.go:1147: Error: Query issue: delete from compliance_operator_rule_v2 where compliance_operator_rule_v2.Id = ANY($1::text[]): conn busy
  ```

• Checked RHACS resource version in the stackrox namespace, which indicates RHACS 4.8.0 is used.

```
$ oc get -n stackrox deploy -o json|jq -c '.items[].spec.template.metadata.labels | {app, "app.kubernetes.io/version"}'|sort -t\" -rk8

{"app":"scanner","app.kubernetes.io/version":"4.8.0"} {"app":"config-controller","app.kubernetes.io/version":"4.8.0"} {"app":"central-db","app.kubernetes.io/version":"4.8.0"} {"app":"central","app.kubernetes.io/version":"4.8.0"} {"app":"admission-control","app.kubernetes.io/version":"4.8.0"}

```

• Checked PVCs, which are bound correctly to PV
  ```
  $ oc get pvc -n stackrox
  NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
  central-db Bound pvc-ac5f2a10-e785-4d28-89f7-65e0006cddcb 300Gi RWO thin-csi <unset> 1y
  central-db-backup Bound pvc-2cb5bbc9-c55a-4883-b1be-8dc0cdff7e25 600Gi RWO thin-csi <unset> 4d
  ```

Slack Discussion Link:

https://redhat-internal.slack.com/archives/C028JE84N59/p1756184061312099