During EIP scale testing, for creating 24000 EIPs on a 120 node baremetal OCP environment, we observed ovnkube-controller CPU usage linearly increased from 4% to 800%. Avergae CPU usage during the test was 360% and max was 1060%. 

Test creates 24000 namespaces wherein for each namespace it creates 1 EIP object with 1 EIP address and 1 Deployment with 1 pod replicas. Thus test creates 24000 namespaces, 24000 EIPs, 24000 pods.

Some customers are using 200 EIPs per node. 8 cores CPU usage on the worker during churn can limit the CPU avaialibity for customer workloads

More details about the testing and observations are at https://docs.google.com/document/d/17NGv6pR-3VFVD5hFzpcdYt5hRZChOSIzmm0BPc-Rdx8/edit?usp=sharing 

Environment details -
OCP deployed on bare metal nodes - 120 workers, 2 infra, 3 masters.
All nodes have same configuration -
CPUs: 80   Memory: 384G   NIC bandwidth: 25 gb/s

    4.16

Always    

Perf & OVN team developed a custom workload for EIP scale testing. We have the OCP deployment now to debug this issue.