During our scale testing, we noticed OVNK annotating master node with the newly created networks, but not deleting them after deleting UDN CRD and namespaces. So these annotations (i.e annonation which start with UDN name, example l3-network-2) exist forever on master node even after deleting UDN and the namespace

oc get nodes ip-10-0-30-56.us-west-2.compute.internal -oyaml

k8s.ovn.org/node-mgmt-port-mac-addresses: '{"default":"72:6d:c7:ad:02:77","l3-network-19":"86:82:bb:aa:44:fa","l3-network-2":"ee:99:2e:bf:90:e2","l3-network-20":"72:3c:2b:9f:75:ae","l3-network-22":"26:15:83:4c:2d:ba","l3-network-23":"1a:78:70:b5:17:54","l3-network-25":"3e:cb:27:73:26:9a","l3-network-27":"f6:a0:75:9e:6a:a5","l3-network-28":"4a:32:83:e7:d7:0f","l3-network-29":"e6:ee:d2:2b:d3:a0","l3-network-3":"5a:8e:9e:2f:25:de","l3-network-30":"16:a6:5d:c7:11:2c","l3-network-31":"2e:72:b6:f3:a5:c3","l3-network-32":"86:4f:9b:f4:12:4c","l3-network-33":"16:ba:fc:97:1a:4b","l3-network-34":"82:ca:3e:4b:a4:ab","l3-network-35":"32:d0:26:e2:29:8f","l3-network-36":"9e:0e:57:c8:5f:55","l3-network-37":"c2:b4:2b:6d:a3:75","l3-network-38":"42:6f:9d:af:16:39","l3-network-39":"2a:30:aa:57:df:69","l3-network-4":"6a:65:92:5e:ba:52","l3-network-40":"9e:5c:d9:14:af:66","l3-network-41":"66:94:d0:e8:db:8b","l3-network-42":"fa:e9:46:6d:a7:2d","l3-network-43":"6a:c4:26:4a:f5:59","l3-network-44":"9a:5b:2d:2e:43:65","l3-network-45":"aa:fa:33:62:e8:43","l3-network-46":"0e:79:fe:e2:f7:7a","l3-network-47":"46:75:51:21:ba:1e","l3-network-48":"46:ee:2d:0d:dd:12","l3-network-49":"06:80:08:3e:49:c5","l3-network-5":"06:5a:01:c6:e0:9d","l3-network-6":"8a:18:a2:8c:6b:1e","l3-network-7":"fa:07:9a:ca:44:fa","l3-network-8":"5a:a3:f9:c6:ee:64","l3-network-9":"c6:b1:1e:b7:b1:e0"}'

Another observation is, OVNK is taking a long time to remove annotations which start with namespaces on master node. For example, below annotation which start with the namespace and UDN combination i.e  "cluster-density-v2-100.l3-network-100":"5e:1a:04:d0:7b:ed" is getting deleted after 1 hour of removing the UDN and namespace 

k8s.ovn.org/node-mgmt-port-mac-addresses: '{"cluster-density-v2-100.l3-network-100":"5e:1a:04:d0:7b:ed",

This is impacting our testing as we can't have back to back runs. We saw this issue on latest nightly as well.