Guide(?) 3scale migration from one OSD cluster to another

MGDAPI team is trying to migrate 3scale from one OSD cluster to another and change the namespace where 3scale will be restored. See this document.

They tried to do a database dump and restore but ended up running into many issues, such as API traffic data not showing in the application, API routes created using the URL from the old cluster, some issues with keycloak and secrets were not valid...they also tried to use 3scale_toolbox but no services were reachable.
(They are using 3scale-operator for installing 3scale.)

The only thing we (3scale) have implemented/documented/testing so far in the life of 3scale is:

• Backup and restore procedures of the main DB and parts of redis
• 3scale_toolbox to copy services (and multiple sub-entities in the data model) from one 3scale service (wherever that may be) to another 3scale service (wherever that maybe).

We have never planned (or done) work to develop a full migration tooling nor have we documented a process for that.

Backup/restore functionality was recommended as a solution to mgdapi needs, but our backup is designed for migration within the same cluster (done and documented by ops team).

This epic is for work to migrate, duplicate, mirror... plus changes (e.g. new domains, etc.)
It will involve applications, keycloak, service names in clusters, routes synced by Zync, domains etc....

Some important points to consider:

• this is far beyond the scope of backup/restore
• this needs product to lead definition work (e.g. applications, OIDC tokens, analytics, etc., included? goal is zero or limited to a specified downtime? to/from accounts on saas/on-prem/managed?)
• this will require toolbox, backend, system, zync, (operator?) involvement