1. Proposed title of this feature request

Reduce memory usage by OVN when 20k+ multinetworkpolicy rules are added to namesapce.

2. What is the nature and description of the request?

Customer is seeing that the OVN components will use about 5+GB or memory on each node when adding multiple multinetworkpolicy objects (with 4k in rules each) to a namespace. This is to much and they would like to see what can be done to reduce the memory usage.

3. Why does the customer need this? (List the business requirements here)

5GB of memory usage on each node is too much

4. List any affected packages or components.

OVN-KUBE SDN 

5. Customer points to memory usage at time of multinetworkpolicy added:
USER PID %CPU %MEM VSZ-MiB RSS-MiB TTY STAT START TIME COMMAND
root 2062807 0.3 0.3 3803 3642 ? - 14:17 0:28 ovn-northd --no-chdir -vconsole:info
root 2062902 0.7 0.2 2253 2248 ? - 14:17 1:03 ovsdb-server -vconsole:info -vfile:off
openvsw+ 4758 2.8 0.0 10948 661 ? - Nov18 76:36 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer
Heap PPROF of OVN Kube controller

 Showing nodes accounting for 1810.40MB, 97.28% of 1861.01MB total
Dropped 235 nodes (cum <= 9.31MB)
flat flat% sum% cum cum%
1329.43MB 71.44% 71.44% 1329.43MB 71.44% encoding/json.(*decodeState).literalStore
256MB 13.76% 85.19% 256MB 13.76% encoding/json.(*Decoder).refill
73.51MB 3.95% 89.14% 73.51MB 3.95% encoding/json.unquote (inline)
54.51MB 2.93% 92.07% 54.51MB 2.93% github.com/ovn-org/ovn-kubernetes/go-controller/pkg/nbdb.copyACLExternalIDs (inline)
24.51MB 1.32% 93.39% 24.51MB 1.32% github.com/ovn-org/ovn-kubernetes/go-controller/pkg/nbdb.copyACLOptions (inline)
 
$ grep peak ovnkube-node-2xtzx_openshift-ovn-kubernetes_northd-c45ed17eb425d864529cb27ba95b2d4cb195190d3086050ca1abc5698e4c903b.log
2024-11-20T14:17:53.968988065+00:00 stderr F 2024-11-20T14:17:53.968Z|00017|memory|INFO|17416 kB peak resident set size after 14.9 seconds
2024-11-20T14:20:32.807116638+00:00 stderr F 2024-11-20T14:20:32.807Z|00019|memory|INFO|peak resident set size grew 2493% in last 158.8 seconds, from 17416 kB to 451592 kB
2024-11-20T14:20:44.928158286+00:00 stderr F 2024-11-20T14:20:44.928Z|00021|memory|INFO|peak resident set size grew 87% in last 12.1 seconds, from 451592 kB to 845832 kB
2024-11-20T15:54:14.323912782+00:00 stderr F 2024-11-20T15:54:14.323Z|00025|memory|INFO|peak resident set size grew 93% in last 5609.4 seconds, from 845832 kB to 1632824 kB
2024-11-20T15:54:48.829733420+00:00 stderr F 2024-11-20T15:54:48.829Z|00042|memory|INFO|peak resident set size grew 100% in last 34.5 seconds, from 1632824 kB to 3273272 kB