We got a OVN-IC V3 image for performance testing. In this image we see very high memory usage by ovnkube-controller when compared to the v0 image which we tested earlier.

Total memory usage by ovn in each worker in v3 image is 5.66 GiB compared to 1.44 GiB in v0 image.
ovnkube-controller (or ncm) memory usage in each worker in v3 image is 3.24 GiB compared to 800 MiB in v0 image.
So there is 4 times increase in memory usage across these two images.
Need to analyse which change introduced this memory regression.

V0 images:
OVN_IMAGE=quay.io/itssurya/dev-images:ic-scale-v0
CNO_IMAGE=quay.io/itssurya/dev-images:ic-cno-hack

V3 images:
OVN_IMAGE=quay.io/itssurya/dev-images:ic-scale-v3 
CNO_IMAGE=quay.io/itssurya/dev-images:ic-scale-cno-v3

This testing is done on 120 node environment (aws) using cluster-density-v2(churn=true, 800 iterations) workload.