When upgrading OpenShift from 4.13 to 4.14, the CNO migrates ovn-ipsec to its new architecture in 4.14 and then just waits until everything else is migrated. This can take a while in a small cluster like the test I did, but mid to large clusters this may mean that ovn-ipsec will stay unavailable with initContainers in crashloopbackoff [1]. This can be concerning to customers and they expect that such disruption to be as little as possible, even though they understand that some is normal.

Looking at the daemonsets the ovn-ipsec pods will need ovnkube-node to configure the node properly in order for the initContainer to create the key pairs and then have ovn-ipsec started. In normal upgrades between errata releases, I understand why we upgrade ipsec first, but in this case I don't think CNO making this upgrade first is not the best procedure.

Important thing I noticed in the upgrade frpom 4.13.38 to 4.14.20, is that there was an intermediate rollout on ovnkube-node and ovnkube-master daemonsets before the actual migration started. During all this time the new ovn-ipsec was already deployed and pods all crashing.

[1] + echo '2024-05-09T11:22:03+00:00 - ERROR - /etc/ovn/ovnkube-node-certs/ovnkube-client-current.pem not found'
2024-05-09T11:22:03+00:00 - ERROR - /etc/ovn/ovnkube-node-certs/ovnkube-client-current.pem not found
+ return 1
/bin/bash: line 16: return: can only `return' from a function or sourced script

    

OCP 4.14
    

Often
    

    1. Install OCP on 4.13
    2. Upgrade to 4.14
    3. Monitor the OVN pods
    

ovn-ipsec pods will stay in CrashLoopBackOff until the rest of the entire OVN stack is migrated and running
    

Minimum disruption problem during the upgrade