Description of problem:

We are running network policy tests on OCP 4.16 ROSA 24 node cluster environment. 

We are deleting all the namespaces once the test is done. However we still see the high memory usage by OVN components after deleting all the resources. We kept cluster idle for 4 hours after deletion and we still observed memory usage has not come down.

Configuration of our test
kube-burner-ocp network-policy --iterations 240 --pods-per-namespace 10 --netpol-per-namespace 50 --pod-selectors 1 --local-pods 10 --single-ports 5 --port-ranges 5 --remotes-namespaces 5 --remotes-pods 5 --cidrs 5

This will create 240 namespaces. Each namespace has 10 pods and 100 network policies (50 ingress network policies and 50 egress network policies).  Each Ingress & egress network policy is configured with 10 local pods, 1 pod selector, 5 remote namespaces and 5 remote pods in the pod selector, 5 single ports and 5 port ranges, 5 cidr rules.

OVN resources -

Average number of OVN resources created per worker node during the test -

ovs flows 1822347

logical flows - 162366

ACL - 290439

port groups - 50629

address set - 887

Average number of OVN resources  per worker node after cleanup (cluster kept idle for 4 hours after the cleanup while taking this measurement)  -

OVS flows 3043

logical flows 2642

ACL - 39

port groups - 231

address set - 215

Memory usage -

NBDB, SBDB, Northd memory usage is same even after the cleanup.

ovnkube-controller came down to 250 MiB after cleanup, but after 1 hour it reached back to 1.35 GiB.

ovs-vswitchd memory usage didn't come down after cleanup.

Description of columns in the below table -

1. Before the test - Memory usage before we created any resources on the env
2. After netpol creation - After test is run which creates all the network policies
3. After cleanup - Once test is finsihed, we deleted all the resources and then measured resource usage.
4. 4 hours after cleanup - we kept the clude idle for 4 hours after the cleanup and measured resource usage

Memory usage by OVN components is as follows

must-gather which is run after 4 hours after the cleanup https://storage.scalelab.redhat.com/anilvenkata/must-gather.local.257822608749119183.tar.gz