Description of problem: When routingViaHost option is modified in network.operator/cluster, ovnkube-node pods are restarted but not ovnkube-control-plane pods. Thus, cluster-manager does not track the change in gateway mode from local to shared and vice-versa. This impacts EgressService creation with sourceIPBy set to Network. We have to manually restart control-plane pods for to ensure EgressService creation.

Version-Release number of selected component (if applicable): 4.18.0-rc9

How reproducible: 100%

Steps to Reproduce:

1. network.operator/cluster configured with routingviaHost=false

2. Create EgressService with sourceIPBy=Network. Notice status.Host is empty.

3. Update network.operator/cluster with routingviaHost=true.

4. ovnkube-node pods gets restarted. wait for it to be stable. Notice gateway mode is changed in the configMap ovnkube-config

5. ovnkube-control-plane pods are not restarted thus  EgressService is still not created (status.Host="") 

6. Manuallt restart control-plane pods. Notice EgressService status.Host is set to ALL.

Actual results: Control-plane does not recognise the gateway mode change. Thus EgressService is not created

Expected results: Control-plane pods identify the change and restart. Thus, EgressService can be handled.

Additional info:

mustgather : https://drive.google.com/file/d/1cxoyYO1Nmj8ILE6PnG25S7JJVWCj8xcv/view?usp=sharing