Because SaaS database is pretty large, every dangerous change in the database needs to be taken care of by operations people. This is a fastidious manual task and error prone.

We need to define how we do SaaS migrations with ops team.

Documentation: https://docs.google.com/document/d/1yc0XsFSZvc5JNxTYW0jEEcHf9w93Wg96KaKm3zMg-ac/edit#

Dev notes

1st phase: Documentation - document what ops is doing now manually

• We already did what we can to have at dev time warning through strong_migrations
• We need to run dangerous migration through pt-osc with downtime sometimes (accounts , proxies or services table touched for example)
• We need to add instructions tested in preview
• Find difference between current SaaS and OCP SaaS
• Make sure whole cluster won't crash
• Have a pod to do the migration
• Do we also provide to customers through job or whatever kubernetes provides?

2nd phase: Automation
Not what we are going to do now, we decide just to document it
Based on those documentation we could create better scripts to automate it and maybe provide those scripts to customers on-premises

This is to make sure we can automate this in OCP and have tools to rollback, etc.

Example:

• a migration on a big table has 2 instructions to add 2 columns
• one instruction goes well
• the other somehow failed because of a connection issue
• there is no way to finalize the migration right now on-premises without removing the first column