RCA request for failed migration

Scenario:
Cu performed the 3scale 2.11 migration from OCP 3.11 to OCP 4.X. Then upgraded with the operator from 3scale 2.11 to 3scale 2.12. After the upgrade, when redeploying system-app, the system-app-hook-pre pod was in CrashLoopBackOff state.

Issue found:

The following error was found in the system-app-hook-pre pod logs:

Migrating to CreateEmailConfigurations (20211215120415)
== 20211215120415 CreateEmailConfigurations: migrating ========================
[strong_migrations] DANGER: Lock timeout is longer than 10 seconds: 31536000
-- adapter_name()
   -> 0.0000s
-- adapter_name()
   -> 0.0000s
-- create_table(:email_configurations, {:id=>:integer})
rake aborted!
StandardError: An error has occurred, all later migrations canceled:

Index name 'index_email_configurations_on_account_id' on table 'email_configurations' already exists

Steps done to understand the issue:
Checked the migrations_table.

mysql> show databases;
mysql> use system;
mysql> select * from schema_migrations;

The migration with version 20211215120415 was missing as well as the subsequent for 3scale 2.12 (20220228215614).

Workaround to unblock the customer:

Asked the Cu to write 20211215120415 to the schema_migrations table and redeploy system-app. The workaround worked and we could verify that the next migration was also added:

Questions:

• Why has this happened?
• How can we prevent this from happening again?