Description of problem:
EgressIP doesn't work on azure hosted cluster
Version-Release number of selected component (if applicable):
4.16.0-0.nightly-2024-09-03-161040

How reproducible:
Always

Steps to Reproduce:
egressIP continually failed on ci https://reportportal-openshift.apps.ocp-c1.prod.psi.redhat.com/ui/#prow/launches/all/606296/82292538?item1[…]252Cti_sok676b1k8j5%252Cti_r1ifkmkw19o1

Then manually reproduced it on 4.16 version

1.

% oc get clusterversion
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.16.0-0.nightly-2024-09-03-161040   True        False         51m     Cluster version is 4.16.0-0.nightly-2024-09-03-161040
% oc get nodes
NAME                                  STATUS   ROLES    AGE   VERSION
44b5d8523d1079fbfd64-f412695a-2zd4g   Ready    worker   52m   v1.29.7+4510e9c
44b5d8523d1079fbfd64-f412695a-p9tkk   Ready    worker   35m   v1.29.7+4510e9c
44b5d8523d1079fbfd64-f412695a-qmg8g   Ready    worker   44m   v1.29.7+4510e9c
% oc get egressIP -o yaml
apiVersion: v1
items:
- apiVersion: k8s.ovn.org/v1
  kind: EgressIP
  metadata:
    creationTimestamp: "2024-09-04T07:54:13Z"
    generation: 2
    name: egressip-2
    resourceVersion: "25280"
    uid: 7536dc5b-3433-44c5-9a79-8d416eb9439e
  spec:
    egressIPs:
    - 10.0.0.100
    namespaceSelector:
      matchLabels:
        name: qe
  status:
    items:
    - egressIP: 10.0.0.100
      node: 44b5d8523d1079fbfd64-f412695a-2zd4g
kind: List
metadata:
  resourceVersion: ""


% oc get cloudprivateipconfig -o yaml
apiVersion: v1
items:
- apiVersion: cloud.network.openshift.io/v1
  kind: CloudPrivateIPConfig
  metadata:
    annotations:
      k8s.ovn.org/egressip-owner-ref: egressip-2
    creationTimestamp: "2024-09-04T07:54:13Z"
    finalizers:
    - cloudprivateipconfig.cloud.network.openshift.io/finalizer
    generation: 1
    name: 10.0.0.100
    resourceVersion: "25279"
    uid: f7a352ab-dfd7-4054-adef-8e67cff3f535
  spec:
    node: 44b5d8523d1079fbfd64-f412695a-2zd4g
  status:
    conditions:
    - lastTransitionTime: "2024-09-04T07:54:20Z"
      message: IP address successfully added
      observedGeneration: 1
      reason: CloudResponseSuccess
      status: "True"
      type: Assigned
    node: 44b5d8523d1079fbfd64-f412695a-2zd4g
kind: List
metadata:
  resourceVersion: ""
% oc get pods -n test -o wide
NAME        READY   STATUS    RESTARTS   AGE     IP            NODE                                  NOMINATED NODE   READINESS GATES
hello-pod   1/1     Running   0          9m50s   10.133.2.19   44b5d8523d1079fbfd64-f412695a-p9tkk   <none>           <none>

Before apply label name=qe to test namespace
% oc rsh -n test hello-pod
~ $ curl -I www.google.com
HTTP/1.1 200 OK
Content-Type: text/html; charset=ISO-8859-1
Content-Security-Policy-Report-Only: object-src 'none';base-uri 'self';script-src 'nonce-6GQ_dKaMJv56tSV4REwdqg' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle.com/csp/gws/other-hp
P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info."
Date: Wed, 04 Sep 2024 07:55:19 GMT
Server: gws
X-XSS-Protection: 0
X-Frame-Options: SAMEORIGIN
Transfer-Encoding: chunked
Expires: Wed, 04 Sep 2024 07:55:19 GMT
Cache-Control: private
Set-Cookie: AEC=AVYB7cooomOER_1unbM8kK3bxAsihbPBJWvoY8o0h9PixKa8-rKZXL_79Mg; expires=Mon, 03-Mar-2025 07:55:19 GMT; path=/; domain=.google.com; Secure; HttpOnly; SameSite=lax
Set-Cookie: NID=517=KqZXdcUCELVo9lig29gIMSdqJolZvyAaV4RRnd1TRhziowfgab-V3QsXLTwi5NkFebmlX4TKUiuwXABAKG4XrK9aQS4JzWBslWjsLsI4_e9Rim8e_IemlImWMnQcB_SLKJ92T7yQ1xmfLVy0qOuSbtCbHhUTy_O4X7v7JYOGA504iX8BELho; expires=Thu, 06-Mar-2025 07:55:19 GMT; path=/; domain=.google.com; HttpOnly

After applying label name=qe to test namespace
2.

3.

Actual results:
% oc rsh -n test hello-pod
~ $
~ $ curl -I www.google.com --connect-timeout 5
curl: (28) Failed to connect to www.google.com port 80 after 4929 ms: Operation timed out
~ $ curl -I www.google.com --connect-timeout 5
curl: (28) Failed to connect to www.google.com port 80 after 4930 ms: Operation timed out
~ $ exit

Expected results:
Egress traffic should not break.

Additional info:

Please fill in the following template while reporting a bug and provide as much relevant information as possible. Doing so will give us the best chance to find a prompt resolution.

Affected Platforms:

Is it an

1. internal CI failure
2. customer issue / SD
3. internal RedHat testing failure

If it is an internal RedHat testing failure:

• Please share a kubeconfig or creds to a live cluster for the assignee to debug/troubleshoot along with reproducer steps (specially if it's a telco use case like ICNI, secondary bridges or BM+kubevirt).

If it is a CI failure:

• Did it happen in different CI lanes? If so please provide links to multiple failures with the same error instance
• Did it happen in both sdn and ovn jobs? If so please provide links to multiple failures with the same error instance
• Did it happen in other platforms (e.g. aws, azure, gcp, baremetal etc) ? If so please provide links to multiple failures with the same error instance
• When did the failure start happening? Please provide the UTC timestamp of the networking outage window from a sample failure run
• If it's a connectivity issue,
• What is the srcNode, srcIP and srcNamespace and srcPodName?
• What is the dstNode, dstIP and dstNamespace and dstPodName?
• What is the traffic path? (examples: pod2pod? pod2external?, pod2svc? pod2Node? etc)

If it is a customer / SD issue:

• Provide enough information in the bug description that Engineering doesn’t need to read the entire case history.
• Don’t presume that Engineering has access to Salesforce.
• Do presume that Engineering will access attachments through supportshell.
• Describe what each relevant attachment is intended to demonstrate (failed pods, log errors, OVS issues, etc).
• Referring to the attached must-gather, sosreport or other attachment, please provide the following details:
  • If the issue is in a customer namespace then provide a namespace inspect.
  • If it is a connectivity issue:
    • What is the srcNode, srcNamespace, srcPodName and srcPodIP?
    • What is the dstNode, dstNamespace, dstPodName and dstPodIP?
    • What is the traffic path? (examples: pod2pod? pod2external?, pod2svc? pod2Node? etc)
    • Please provide the UTC timestamp networking outage window from must-gather
    • Please provide tcpdump pcaps taken during the outage filtered based on the above provided src/dst IPs
  • If it is not a connectivity issue:
    • Describe the steps taken so far to analyze the logs from networking components (cluster-network-operator, OVNK, SDN, openvswitch, ovs-configure etc) and the actual component where the issue was seen based on the attached must-gather. Please attach snippets of relevant logs around the window when problem has happened if any.

• When showing the results from commands, include the entire command in the output.  
• For OCPBUGS in which the issue has been identified, label with “sbr-triaged”
• For OCPBUGS in which the issue has not been identified and needs Engineering help for root cause, label with “sbr-untriaged”
• Do not set the priority, that is owned by Engineering and will be set when the bug is evaluated
• Note: bugs that do not meet these minimum standards will be closed with label “SDN-Jira-template”
• For guidance on using this template please see
  OCPBUGS Template Training for Networking  components