Today, when OpenShift is configured with storage layer encryption, we have a process to automate key rotation every week. Part of this process is a complete storage migration from the old key to the new one, which requires to re-encrypt all the previously encrypted resources. This can put the system under some level of stress and makes key rotation a long process.

Instead of that, we could just do "soft migration" to the new key meaning that any new/updated resource would be encrypted with the new key but existing object would still be encrypted with the previous one. This ensure that we have a well-honed system in place for rotations without putting any burden on the cluster.
The only problem would be that if one old key leaks, some part of the data stored in etcd would be at risk. But this is not something that can be automated since there is no way for a system to be aware of leaks. So, for these scenarios we should leave the possibility to users to manually launch an "hard migration" from the old keys to a new one so that all the data at risk will get re-encrypted with a new key.

Slack discussion: https://redhat-internal.slack.com/archives/CC3CZCQHM/p1675963989835079

Notes:

• Disable full migration (or doing it on bi-yearly basis)
• Still do key rotation weekly
• Document how admin can trigger the migration manually (in case leaked key)
• Create the look-up key func for for provider/transformer key encryption
• xref: https://issues.redhat.com/browse/OCPBUGS-7890 (create first class mechanism to force rotate encryption)