Description of problem:

We are running node-density-heavy workload on a 120 node OVN-IC environment. OVN components are using nearly 1 core (93%) in each worker node in OVN-IC. In that, ovnkube-network-controller-manager is using 22% CPU.

ovnkube-master is using lesser (or similar) i.e 18% in master node in classis OVN for the same workload. ovnkube-ncm has to be improved to use lesser cpu as it is handling only local pods, which will help in reducing overall CPU usage by OVN components in the worker node (i.e bringing down 93% usage).

Please check https://docs.google.com/document/d/1t5pLoICUFsYo_7EMJ1tZk4tTxavUqRaKFjlEzIhwBnQ/edit?usp=sharing for more details

Version-Release number of selected component (if applicable):

OCP 4.14
OVN image: quay.io/itssurya/dev-images:ic-scale-v0
CNO image: quay.io/itssurya/dev-images:ic-cno-hack

How reproducible:

Steps to Reproduce:

1. Deploy 120 node OCP environment on AWS
2. Apply OVN builds
3. Run node-denisty-heavy workload  

Actual results:

Expected results:

Additional info: