Description of problem: [BGP EIP on default network pre-merge testing] legacy egressIP (an IP address from same subnet of egress node) is not assigned to egress node

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

0. With basic BGP route advertisement on default network enabled and working

1. label a node to be egress node with egress-assignable=true label

2. create a namespace, label the namespace with a label that matches namespaceSelector of egressIP object that will be created in step 3

3. choose an unused IP from same subnet of egress node, use it in egressIP object, create egressIP object

1. oc get egressip NAME EGRESSIPS ASSIGNED NODE ASSIGNED EGRESSIPS

egressip1 192.168.111.201

egressip2 8.8.8.8 worker-2 8.8.8.8

Actual results:  egressIP from same subnet is not assigned to egress node

Expected results: egressIP from same subnet should be assigned to egress node

Additional info:

ovnkube-node log:  https://drive.google.com/file/d/1FvjzL6Vy17beMqdXP0ruXNxWQdUA1-Zw/view?usp=drive_link

must-gather: https://drive.google.com/file/d/1x0sXzQLw46_2W46O4J0bGDqTHfYEnt-x/view?usp=drive_link

When I was trying to clean up configuration after the test, I first deleted the namespace, then I found the legacy egressIP is assigned to egress node, but the second egressIP with random IP got unassigned.    I added the namespace back and re-label it, then I saw both egressIP assigned.

[root@sdn-09 log_network_logs]# oc delete ns test
namespace "test" deleted
[root@sdn-09 log_network_logs]# oc get egressip
NAME        EGRESSIPS         ASSIGNED NODE   ASSIGNED EGRESSIPS
egressip1   192.168.111.201   worker-2        192.168.111.201
egressip2   8.8.8.8                           

[root@sdn-09 log_network_logs]# oc new-project test

[root@sdn-09 log_network_logs]# oc get egressip
NAME        EGRESSIPS         ASSIGNED NODE   ASSIGNED EGRESSIPS
egressip1   192.168.111.201   worker-2        192.168.111.201
egressip2   8.8.8.8           worker-2        8.8.8.8

Now, both egressIPs are advertised:

[root@sdn-09 demo]# sudo podman exec -it 5eda0c5903e6 /bin/sh
/ # vtysh
% Can't open configuration file /etc/frr/vtysh.conf due to 'No such file or directory'.
Configuration file[/etc/frr/frr.conf] processing failure: 11

Hello, this is FRRouting (version 9.1.2_git).
Copyright 1996-2005 Kunihiro Ishiguro, et al.

sdn-09.ocpnet.eng.rdu2.dc.redhat.com# show ip bgp
BGP table version is 12, local router ID is 192.168.122.1, vrf id 0
Default local pref 100, local AS 64512
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

    Network          Next Hop            Metric LocPrf Weight Path
 *>i8.8.8.8/32       192.168.111.25           0    100      0 i
 *>i10.128.0.0/23    192.168.111.20           0    100      0 i
 *>i10.128.2.0/23    192.168.111.24           0    100      0 i
 *>i10.129.0.0/23    192.168.111.21           0    100      0 i
 *>i10.129.2.0/23    192.168.111.25           0    100      0 i
 *>i10.130.0.0/23    192.168.111.22           0    100      0 i
 *>i10.131.0.0/23    192.168.111.23           0    100      0 i
 *> 192.168.1.0/24   0.0.0.0                  0         32768 i
 *>i192.168.111.201/32
                    192.168.111.25           0    100      0 i
 *> 192.169.1.1/32   0.0.0.0                  0         32768 i

Displayed  10 routes and 10 total paths

Please fill in the following template while reporting a bug and provide as much relevant information as possible. Doing so will give us the best chance to find a prompt resolution.

Affected Platforms:

Is it an

1. internal CI failure
2. customer issue / SD
3. internal RedHat testing failure

If it is an internal RedHat testing failure:

• Please share a kubeconfig or creds to a live cluster for the assignee to debug/troubleshoot along with reproducer steps (specially if it's a telco use case like ICNI, secondary bridges or BM+kubevirt).

If it is a CI failure:

• Did it happen in different CI lanes? If so please provide links to multiple failures with the same error instance
• Did it happen in both sdn and ovn jobs? If so please provide links to multiple failures with the same error instance
• Did it happen in other platforms (e.g. aws, azure, gcp, baremetal etc) ? If so please provide links to multiple failures with the same error instance
• When did the failure start happening? Please provide the UTC timestamp of the networking outage window from a sample failure run
• If it's a connectivity issue,
• What is the srcNode, srcIP and srcNamespace and srcPodName?
• What is the dstNode, dstIP and dstNamespace and dstPodName?
• What is the traffic path? (examples: pod2pod? pod2external?, pod2svc? pod2Node? etc)

If it is a customer / SD issue:

• Provide enough information in the bug description that Engineering doesn’t need to read the entire case history.
• Don’t presume that Engineering has access to Salesforce.
• Do presume that Engineering will access attachments through supportshell.
• Describe what each relevant attachment is intended to demonstrate (failed pods, log errors, OVS issues, etc).
• Referring to the attached must-gather, sosreport or other attachment, please provide the following details:
  • If the issue is in a customer namespace then provide a namespace inspect.
  • If it is a connectivity issue:
    • What is the srcNode, srcNamespace, srcPodName and srcPodIP?
    • What is the dstNode, dstNamespace, dstPodName and dstPodIP?
    • What is the traffic path? (examples: pod2pod? pod2external?, pod2svc? pod2Node? etc)
    • Please provide the UTC timestamp networking outage window from must-gather
    • Please provide tcpdump pcaps taken during the outage filtered based on the above provided src/dst IPs
  • If it is not a connectivity issue:
    • Describe the steps taken so far to analyze the logs from networking components (cluster-network-operator, OVNK, SDN, openvswitch, ovs-configure etc) and the actual component where the issue was seen based on the attached must-gather. Please attach snippets of relevant logs around the window when problem has happened if any.

• When showing the results from commands, include the entire command in the output.  
• For OCPBUGS in which the issue has been identified, label with “sbr-triaged”
• For OCPBUGS in which the issue has not been identified and needs Engineering help for root cause, label with “sbr-untriaged”
• Do not set the priority, that is owned by Engineering and will be set when the bug is evaluated
• Note: bugs that do not meet these minimum standards will be closed with label “SDN-Jira-template”
• For guidance on using this template please see
  OCPBUGS Template Training for Networking  components