We run various OpenShift clusters with OVN-Kubernetes as CNI, and use netpol and egressfirewall massively.

In the design of OVN-Kubernetes, once an ovnkube-master is elected leader, it starts watching all pods and netpols of the cluster and configuring ovn db. During that time (which can take up to 1 hour), new pods created will start without having their proper connectivity  (they see traffic dropped by a default deny egress acl), until ovnkube-master "catches up" and updates the needed ovn resources.

There is a vicious effect when performing rolling updates (for example, OCP upgrades, machineconfig deployments on both master and workers, kubeletconfig changes, etc.) as there could be an election of a new ovnkube-master while the previous one did not finish its sync yet, and pods on worker nodes get drained and recreated, amplifying the problem greatly.

The current focus has been to improve the time to perform the sync (what is measured by the ovnkube_master_ready_duration_seconds) and to workaround the issue by pausing the worker mcp in this fashion:
1- pause the worker mcp
2- run the upgrade
3- wait until the 3 masters rebooted
4- wait until max(ovnkube_master_ready_duration_seconds) > 0
5- unpause the worker mcp

However it is hard to ensure that this process (manual) is followed for every change we do that involve updating all mcp. And the improvement (that we expect in 4.14 and hope to get in 4.13) only reduces the problem but does not fix it entirely. It will still be possible to have pods started without connectivity, even with 4.14, as the readiness ovnkube-master is never guaranteed.

We need a solution to prevent the disruption of the network traffic of pods started during ovnk synchro.

One thing that was suggested to us is to use some kind of canary rollout (https://docs.openshift.com/container-platform/4.10/updating/update-using-custom-machine-config-pools.html#update-using-custom-machine-config-pools-mcp-remove_update-using-custom-machine-config-pools) but this process is also very manual today. One thing that would work for us is to have an automated canary rollout, where the mcp of the workers is only started after a canary node is "fine" - and we could implement custom checks, such as a check of ovnkube-master readiness, on the canary.

4.10.52, 4.12.19,4.13.3

1.
2.
3.