[OVN-IC] High CPU usage of ovnkube-node on the worker nodes during resource deletion

We are running a test that has two distinct phases - a creation/ramp-up phase that creates a lot of resources like pods, services, network policies, namespaces etc and then a churn phase that deletes and recreates 10% of all resources created for about an hour to mimic a cluster in steady state. We notice that ovnkube-node container on the worker nodes is very spiky during the deletion phase as compared to the create phase in OVN-IC. We see a lot more CPU usage during the deletion actions as part of the "churn" phase when compared to the creation actions in the ramp-up phase.

4.14.0-0.nightly-2023-08-08-094653

100%

1. Use kube-burner to run this workload on a ROSA/Self-managed OCP cluster
2. https://github.com/smalleni/kube-burner/commit/d2b4f20f1de20ca0e70d71070331ae61e15698a0 
3. kube-burner ocp cluster-density-v2 --iterations=650

High CPU usage during the churn (resource deletion/re-creation) phase

CPU usage should be lower

The biggest difference in CPU usage when comparing the ovnkube-node pod seems to be coming from the CPU usage of the ovnkube-node container during creation and deletion. The other containers are not contributing much to the difference.