[OVN-IC] Regression in Ovnkube-node container memory usage (380% increase)

Ovnkube-node container max memory usage was 110 MiB with 4.14.0-0.nightly-2023-05-18-231932 image and now it is 530 MiB with 4.14.0-0.nightly-2023-07-31-181848 image, for the same test (cluster-density-v2 with 800 iterations, churn=false) on 120 node environment. We observed the same pattern in the OVN-IC environment as well.

Note: As churn is false, we are calculating memory usage for only resource creation.

Grafana panel for OVN with 4.14.0-0.nightly-2023-05-18-231932 image -

https://grafana.rdu2.scalelab.redhat.com:3000/dashboard/snapshot/H9pAb07fsPEOFyd5dhKLFP602A7S18uC

Grafana panel for OVN with 4.14.0-0.nightly-2023-07-31-181848 image -

https://grafana.rdu2.scalelab.redhat.com:3000/dashboard/snapshot/8158bJgv3e4P2uiVernbc2E5ypBWFYHt

As the test was successfully run in the CI, we couldn't collect a must-gather. I can provide must-gather and pprof data if needed.

We observed 100 MiB to 550 MiB increase in OVN-IC between 4.14.0-0.nightly-2023-06-12-141936 and  4.14.0-0.nightly-2023-07-30-191504 versions.

OVN-IC  4.14.0-0.nightly-2023-06-12-141936

https://grafana.rdu2.scalelab.redhat.com:3000/dashboard/snapshot/o5SXLdHIL8whsdgaMyXwWamipBP8J2fF

OVN-IC 4.14.0-0.nightly-2023-07-30-191504

https://grafana.rdu2.scalelab.redhat.com:3000/dashboard/snapshot/NMuSQx7YAJ9jokoKMl6Me9StHp33tjwD