Description of problem:

When a primary UDN or CUDN is created, it creates what is known as a secondary zone network controller that handles configuring OVN and getting the network created so that pods can be attached. The time it takes for this to happen can be up to a minute if namespaces are being deleted on the cluster while the UDN controller is starting.

This is because if the namespace is deleted, GetActiveNetworkForNamespace will fail, and the pod will be retried for up to a minute.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

1. Create a bunch of namespaces with pods

2. Create a primary CUDN or UDN

3. Very quickly start deleting namespaces that have pods in them while OVNK is starting up its network controller

Actual results:

Logs show time to start the controller taking a minute:
I0131 16:15:30.383221 5583 secondary_layer2_network_controller.go:365] Starting controller for secondary network e2e-network-segmentation-e2e-8086.blue I0131 16:16:30.390813 5583 secondary_layer2_network_controller.go:369] Starting controller for secondary network e2e-network-segmentation-e2e-8086.blue took 1m0.007579788s

Expected results:

Once started the controller should only take a few seconds (depending on cluster size and load) to finish starting.