Description of problem:  [OVNK BGP]After removing then adding back a node, BGP neighbor was formed for new node, but its ip route is not added back into ip route table

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

1. Enabled featuregate and routeAdvertisements through FRR by patching CNO.  Created a external FRR container as external router to cluster nodes, applied receive_all.yaml and routeAdvertisements.yaml.  Verified all nodes formed neighbor with external router, and ip routes of all nodes are showing on external router

1. cat receive_all.yaml 
   apiVersion: frrk8s.metallb.io/v1beta1
   kind: FRRConfiguration
   metadata:
     name: receive-all
     namespace: openshift-frr-k8s
   spec:
     bgp:
       routers:
       - asn: 64512
         neighbors:
         - address: 192.168.122.1
           asn: 64512
           toReceive:
             allowed:
               mode: all

1. cat RouteAdvertisements.yaml
   apiVersion: k8s.ovn.org/v1
   kind: RouteAdvertisements
   metadata:
     name: default
   spec:
       advertisements:
         egressIP: true
         podNetwork: true

2.  Saved one node worker-2's yaml, deleted the node, waited till its ip route is removed from external router

3. Added the node worker-2 back, checked BGP neighbor formation, and ip route table

1. oc get node -owide
   NAME                                       STATUS   ROLES                  AGE     VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE                                                KERNEL-VERSION                 CONTAINER-RUNTIME
   master-0                                   Ready    control-plane,master   3h41m   v1.31.1   192.168.111.20   <none>        Red Hat Enterprise Linux CoreOS 418.94.202410211616-0   5.14.0-427.42.1.el9_4.x86_64   cri-o://1.31.1-5.rhaos4.18.git9704b0c.el9
   master-1                                   Ready    control-plane,master   3h42m   v1.31.1   192.168.111.21   <none>        Red Hat Enterprise Linux CoreOS 418.94.202410211616-0   5.14.0-427.42.1.el9_4.x86_64   cri-o://1.31.1-5.rhaos4.18.git9704b0c.el9
   master-2                                   Ready    control-plane,master   3h41m   v1.31.1   192.168.111.22   <none>        Red Hat Enterprise Linux CoreOS 418.94.202410211616-0   5.14.0-427.42.1.el9_4.x86_64   cri-o://1.31.1-5.rhaos4.18.git9704b0c.el9
   openshift-qe-025.lab.eng.rdu2.redhat.com   Ready    sriov,worker           162m    v1.31.1   192.168.111.26   <none>        Red Hat Enterprise Linux CoreOS 418.94.202410211616-0   5.14.0-427.42.1.el9_4.x86_64   cri-o://1.31.1-5.rhaos4.18.git9704b0c.el9
   openshift-qe-029.lab.eng.rdu2.redhat.com   Ready    sriov,worker           163m    v1.31.1   192.168.111.58   <none>        Red Hat Enterprise Linux CoreOS 418.94.202410211616-0   5.14.0-427.42.1.el9_4.x86_64   cri-o://1.31.1-5.rhaos4.18.git9704b0c.el9
   worker-0                                   Ready    worker                 3h21m   v1.31.1   192.168.111.23   <none>        Red Hat Enterprise Linux CoreOS 418.94.202410211616-0   5.14.0-427.42.1.el9_4.x86_64   cri-o://1.31.1-5.rhaos4.18.git9704b0c.el9
   worker-1                                   Ready    worker                 3h21m   v1.31.1   192.168.111.24   <none>        Red Hat Enterprise Linux CoreOS 418.94.202410211616-0   5.14.0-427.42.1.el9_4.x86_64   cri-o://1.31.1-5.rhaos4.18.git9704b0c.el9
   worker-2                                   Ready    worker                 68m     v1.31.1   192.168.111.25   <none>        Red Hat Enterprise Linux CoreOS 418.94.202410211616-0   5.14.0-427.42.1.el9_4.x86_64   cri-o://1.31.1-5.rhaos4.18.git9704b0c.el9

1. oc get pod -n openshift-frr-k8s -owide
   NAME                                     READY   STATUS    RESTARTS      AGE    IP               NODE                                       NOMINATED NODE   READINESS GATES
   frr-k8s-8csnm                            6/6     Running   6             106m   192.168.111.24   worker-1                                   <none>           <none>
   frr-k8s-8gh98                            6/6     Running   0             69m    192.168.111.25   worker-2                                   <none>           <none>
   frr-k8s-bm2cg                            6/6     Running   6             106m   192.168.111.21   master-1                                   <none>           <none>
   frr-k8s-ff8xf                            6/6     Running   7 (95m ago)   106m   192.168.111.26   openshift-qe-025.lab.eng.rdu2.redhat.com   <none>           <none>
   frr-k8s-fxgwm                            6/6     Running   6             106m   192.168.111.22   master-2                                   <none>           <none>
   frr-k8s-hdddn                            6/6     Running   0             106m   192.168.111.23   worker-0                                   <none>           <none>
   frr-k8s-k7qw4                            6/6     Running   0             102m   192.168.111.58   openshift-qe-029.lab.eng.rdu2.redhat.com   <none>           <none>
   frr-k8s-webhook-server-b8455c4d8-zt2jf   1/1     Running   0             106m   10.131.0.6       worker-0                                   <none>           <none>
   frr-k8s-wjvgh                            6/6     Running   6             106m   192.168.111.20   master-0                                   <none>           <none>
   [root@openshift-qe-026 ~]# oc -c frr rsh frr-k8s-8gh98 -n openshift-frr-k8s -owide
   2024-10-22T17:24:52.124225Z: 2024-10-22T17:24:52.123886Z: executable file `-n` not found in $PATH: No such file or directory

command terminated with exit code 255

On frr container of worker-2

[root@openshift-qe-026 ~]# oc project openshift-frr-k8s 
Already on project "openshift-frr-k8s" on server "https://api.offload.openshift-qe.sdn.com:6443".
[root@openshift-qe-026 ~]# oc -c frr rsh frr-k8s-8gh98
sh-5.1# vtysh

Hello, this is FRRouting (version 8.5.3).
Copyright 1996-2005 Kunihiro Ishiguro, et al.

worker-2# 
worker-2# 
worker-2# show bgp neighbor
BGP neighbor is 192.168.122.1, remote AS 64512, local AS 64512, internal link
  Local Role: undefined
  Remote Role: undefined
Hostname: openshift-qe-026.lab.eng.rdu2.redhat.com
  BGP version 4, remote router ID 192.168.122.1, local router ID 192.168.111.25
  BGP state = Established, up for 01:10:08
  Last read 00:00:08, Last write 00:00:08
  Hold time is 180 seconds, keepalive interval is 60 seconds
  Configured hold time is 180 seconds, keepalive interval is 60 seconds
  Configured conditional advertisements interval is 60 seconds
  Neighbor capabilities:
    4 Byte AS: advertised and received
    Extended Message: advertised and received
    AddPath:
      IPv4 Unicast: RX advertised and received
      IPv6 Unicast: RX advertised
    Long-lived Graceful Restart: advertised and received
      Address families by peer:
    Route refresh: advertised and received(new)
    Enhanced Route Refresh: advertised and received
    Address Family IPv4 Unicast: advertised and received
    Address Family IPv6 Unicast: advertised
    Hostname Capability: advertised (name: worker-2,domain name: n/a) received (name: openshift-qe-026.lab.eng.rdu2.redhat.com,domain name: n/a)
    Graceful Restart Capability: advertised and received
      Remote Restart timer is 120 seconds
      Address families by peer:
        none
  Graceful restart information:
    End-of-RIB send: IPv4 Unicast
    End-of-RIB received: IPv4 Unicast
    Local GR Mode: Helper*

    Remote GR Mode: Helper

    R bit: False
    N bit: True
    Timers:
      Configured Restart Time(sec): 120
      Received Restart Time(sec): 120
    IPv4 Unicast:
      F bit: False
      End-of-RIB sent: Yes
      End-of-RIB sent after update: Yes
      End-of-RIB received: Yes
      Timers:
        Configured Stale Path Time(sec): 360
    IPv6 Unicast:
      F bit: False
      End-of-RIB sent: No
      End-of-RIB sent after update: No
      End-of-RIB received: No
      Timers:
        Configured Stale Path Time(sec): 360
  Message statistics:
    Inq depth is 0
    Outq depth is 0
                         Sent       Rcvd
    Opens:                  1          1
    Notifications:          0          0
    Updates:                1          3
    Keepalives:            71         71
    Route Refresh:          2          2
    Capability:             0          0
    Total:                 75         77
  Minimum time between advertisement runs is 0 seconds

 For address family: IPv4 Unicast
  Update group 1, subgroup 1
  Packet Queue length 0
  Community attribute sent to this neighbor(all)
  Inbound path policy configured
  Outbound path policy configured
  Route map for incoming advertisements is *192.168.122.1-in
  Route map for outgoing advertisements is *192.168.122.1-out
  2 accepted prefixes

 For address family: IPv6 Unicast
  Not part of any update group
  Community attribute sent to this neighbor(all)
  Inbound path policy configured
  Outbound path policy configured
  Route map for incoming advertisements is *192.168.122.1-in
  Route map for outgoing advertisements is *192.168.122.1-out
  0 accepted prefixes

  Connections established 1; dropped 0
  Last reset 01:10:13,  Waiting for peer OPEN
  Internal BGP neighbor may be up to 255 hops away.
Local host: 192.168.111.25, Local port: 53504
Foreign host: 192.168.122.1, Foreign port: 179
Nexthop: 192.168.111.25
Nexthop global: fe80::22f:10ff:fe82:6b16
Nexthop local: fe80::22f:10ff:fe82:6b16
BGP connection: non shared network
BGP Connect Retry Timer in Seconds: 120
Estimated round trip time: 0 ms
Read thread: on  Write thread: on  FD used: 22

worker-2# show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

K>* 0.0.0.0/0 [0/48] via 192.168.111.1, br-ex, src 192.168.111.25, 01:11:43
K>* 10.128.0.0/14 [0/0] via 10.129.2.1, ovn-k8s-mp0, 01:11:39
C>* 10.129.2.0/23 is directly connected, ovn-k8s-mp0, 01:11:39
C>* 169.254.0.0/17 is directly connected, br-ex, 01:11:43
K>* 169.254.0.1/32 [0/0] is directly connected, br-ex, 01:11:43
K>* 169.254.0.3/32 [0/0] via 10.129.2.1, ovn-k8s-mp0, 01:11:39
C>* 172.22.0.0/24 is directly connected, enp1s0, 01:11:43
K>* 172.30.0.0/16 [0/0] via 169.254.0.4, br-ex, src 169.254.0.2, 01:11:43
B>  192.168.1.0/24 [200/0] via 192.168.122.1 (recursive), weight 1, 01:11:27
  *                          via 192.168.111.1, br-ex, weight 1, 01:11:27
C>* 192.168.111.0/24 is directly connected, br-ex, 01:11:43
B>  192.169.1.1/32 [200/0] via 192.168.122.1 (recursive), weight 1, 01:11:27
  *                          via 192.168.111.1, br-ex, weight 1, 01:11:27

On external router:

1. sudo podman exec -it c9cb6d3285b7 /bin/sh
   / # vtysh
   % Can't open configuration file /etc/frr/vtysh.conf due to 'No such file or directory'.
   Configuration file[/etc/frr/frr.conf] processing failure: 11

Hello, this is FRRouting (version 9.1_git).
Copyright 1996-2005 Kunihiro Ishiguro, et al.

openshift-qe-026.lab.eng.rdu2.redhat.com# 
openshift-qe-026.lab.eng.rdu2.redhat.com# 
openshift-qe-026.lab.eng.rdu2.redhat.com# show bgp neighbor

.....

.....

.....

BGP neighbor is 192.168.111.25, remote AS 64512, local AS 64512, internal link
  Local Role: undefined
  Remote Role: undefined
Hostname: worker-2
  BGP version 4, remote router ID 192.168.111.25, local router ID 192.168.122.1
  BGP state = Established, up for 01:12:40
  Last read 00:00:40, Last write 00:00:40
  Hold time is 180 seconds, keepalive interval is 60 seconds
  Configured hold time is 180 seconds, keepalive interval is 60 seconds
  Configured tcp-mss is 0, synced tcp-mss is 1448
  Configured conditional advertisements interval is 60 seconds
  Neighbor capabilities:
    4 Byte AS: advertised and received
    Extended Message: advertised and received
    AddPath:
      IPv4 Unicast: RX advertised and received
    Long-lived Graceful Restart: advertised and received
      Address families by peer:
    Route refresh: advertised and received
    Enhanced Route Refresh: advertised and received
    Address Family IPv4 Unicast: advertised and received
    Address Family IPv6 Unicast: received
    Hostname Capability: advertised (name: openshift-qe-026.lab.eng.rdu2.redhat.com,domain name: n/a) received (name: worker-2,domain name: n/a)
    Version Capability: advertised software version (FRRouting/9.1_git) not received
    Graceful Restart Capability: advertised and received
      Remote Restart timer is 120 seconds
      Address families by peer:
        none
  Graceful restart information:
    End-of-RIB send: IPv4 Unicast
    End-of-RIB received: IPv4 Unicast
    Local GR Mode: Helper*

    Remote GR Mode: Helper

    R bit: True
    N bit: False
    Timers:
      Configured Restart Time(sec): 120
      Received Restart Time(sec): 120
      Configured LLGR Stale Path Time(sec): 0
    IPv4 Unicast:
      F bit: False
      End-of-RIB sent: Yes
      End-of-RIB sent after update: Yes
      End-of-RIB received: Yes
      Timers:
        Configured Stale Path Time(sec): 360
        LLGR Stale Path Time(sec): 0
  Message statistics:
    Inq depth is 0
    Outq depth is 0
                         Sent       Rcvd
    Opens:                  2          2
    Notifications:          0          0
    Updates:                6          3
    Keepalives:            80         80
    Route Refresh:          4          4
    Capability:             0          0
    Total:                 92         89
  Minimum time between advertisement runs is 0 seconds

 For address family: IPv4 Unicast
  Update group 1, subgroup 2
  Packet Queue length 0
  NEXT_HOP is always this router
  Community attribute sent to this neighbor(all)
  0 accepted prefixes

  Connections established 2; dropped 1
  Last reset 01:13:23,   No AFI/SAFI activated for peer (n/a)
  Internal BGP neighbor may be up to 255 hops away.
Local host: 192.168.122.1, Local port: 179
Foreign host: 192.168.111.25, Foreign port: 53504
Nexthop: 192.168.122.1
Nexthop global: ::
Nexthop local: ::
BGP connection: shared network
BGP Connect Retry Timer in Seconds: 120
Estimated round trip time: 1 ms
Read thread: on  Write thread: on  FD used: 31

.....

.....

.....

openshift-qe-026.lab.eng.rdu2.redhat.com# show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

K * 0.0.0.0/0 [0/425] via 255.255.255.0, offloadpr, 01:22:19
K>* 0.0.0.0/0 [0/101] via 10.8.31.254, eno1, 01:22:19
C>* 10.8.0.0/19 is directly connected, eno1, 01:22:19
C>* 10.88.0.0/16 is directly connected, cni-podman0, 01:22:19
B>* 10.128.0.0/23 [200/0] via 192.168.111.20, offloadbm, weight 1, 01:20:18
B>* 10.128.2.0/23 [200/0] via 192.168.111.24, offloadbm, weight 1, 01:20:18
B>* 10.129.0.0/23 [200/0] via 192.168.111.21, offloadbm, weight 1, 01:20:17
B>* 10.130.0.0/23 [200/0] via 192.168.111.22, offloadbm, weight 1, 01:20:17
B>* 10.130.2.0/23 [200/0] via 192.168.111.58, offloadbm, weight 1, 01:20:18
B>* 10.131.0.0/23 [200/0] via 192.168.111.23, offloadbm, weight 1, 01:20:18
B>* 10.131.2.0/23 [200/0] via 192.168.111.26, offloadbm, weight 1, 01:20:17
C>* 172.22.0.0/24 is directly connected, offloadpr, 01:22:19
C>* 192.168.111.0/24 is directly connected, offloadbm, 01:22:19
K>* 255.255.255.0/32 [0/425] is directly connected, offloadpr, 01:22:19

Actual results:  BGP neighbor is formed between new node and external router, but its route 192.168.111.25 is not added back to routing able

Expected results: BGP neighbor is formed between new node and external router, but its route 192.168.111.25 is added back to routing able

Additional info:

must-gather is collected: https://drive.google.com/file/d/1iI9dhllT4aYoQHpZv_7EZ2VxHKTiA1h2/view?usp=drive_link

Please fill in the following template while reporting a bug and provide as much relevant information as possible. Doing so will give us the best chance to find a prompt resolution.

Affected Platforms:

Is it an

1. internal CI failure
2. customer issue / SD
3. internal RedHat testing failure

If it is an internal RedHat testing failure:

• Please share a kubeconfig or creds to a live cluster for the assignee to debug/troubleshoot along with reproducer steps (specially if it's a telco use case like ICNI, secondary bridges or BM+kubevirt).

If it is a CI failure:

• Did it happen in different CI lanes? If so please provide links to multiple failures with the same error instance
• Did it happen in both sdn and ovn jobs? If so please provide links to multiple failures with the same error instance
• Did it happen in other platforms (e.g. aws, azure, gcp, baremetal etc) ? If so please provide links to multiple failures with the same error instance
• When did the failure start happening? Please provide the UTC timestamp of the networking outage window from a sample failure run
• If it's a connectivity issue,
• What is the srcNode, srcIP and srcNamespace and srcPodName?
• What is the dstNode, dstIP and dstNamespace and dstPodName?
• What is the traffic path? (examples: pod2pod? pod2external?, pod2svc? pod2Node? etc)

If it is a customer / SD issue:

• Provide enough information in the bug description that Engineering doesn’t need to read the entire case history.
• Don’t presume that Engineering has access to Salesforce.
• Do presume that Engineering will access attachments through supportshell.
• Describe what each relevant attachment is intended to demonstrate (failed pods, log errors, OVS issues, etc).
• Referring to the attached must-gather, sosreport or other attachment, please provide the following details:
  • If the issue is in a customer namespace then provide a namespace inspect.
  • If it is a connectivity issue:
    • What is the srcNode, srcNamespace, srcPodName and srcPodIP?
    • What is the dstNode, dstNamespace, dstPodName and dstPodIP?
    • What is the traffic path? (examples: pod2pod? pod2external?, pod2svc? pod2Node? etc)
    • Please provide the UTC timestamp networking outage window from must-gather
    • Please provide tcpdump pcaps taken during the outage filtered based on the above provided src/dst IPs
  • If it is not a connectivity issue:
    • Describe the steps taken so far to analyze the logs from networking components (cluster-network-operator, OVNK, SDN, openvswitch, ovs-configure etc) and the actual component where the issue was seen based on the attached must-gather. Please attach snippets of relevant logs around the window when problem has happened if any.

• When showing the results from commands, include the entire command in the output.  
• For OCPBUGS in which the issue has been identified, label with “sbr-triaged”
• For OCPBUGS in which the issue has not been identified and needs Engineering help for root cause, label with “sbr-untriaged”
• Do not set the priority, that is owned by Engineering and will be set when the bug is evaluated
• Note: bugs that do not meet these minimum standards will be closed with label “SDN-Jira-template”
• For guidance on using this template please see
  OCPBUGS Template Training for Networking  components