Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-20501

When updating lots of services, metallb sessions flap between established, active and idle

XMLWordPrintable

    • No
    • CNF Network Sprint 243, CNF Network Sprint 244, CNF Network Sprint 245
    • 3
    • False
    • Hide

      None

      Show
      None

      When updating lots of services, metallb sessions flap between established, active and idle

      See instructions in the folder shared in the private comment.

      You can reproduce this with:

      for f in _output/*; do         oc apply -f $f &       done
      

      and the files that I shared.

      However, the oc apply does not have to run in the background. You can also just run it with:

      for f in _output/*; do         oc apply -f $f ;       done
      

      and the result / impact on metallb and frr will be the same, the sessions flap for a long time.

      This gets particularly bad when deleting a few services - when sequentially deleting a few services, about 1 per second, FRR just goes into "active" state and bgp sessions are down / active for 40 seconds in my test:

      for f in _output/*; do         oc delete -f $f ; sleep 1;       done
      

      Yields on the peers (example output from one of them):

       while true; do sleep 2 ; echo =========================; date; vtysh -c 'show ip bgp summ'; vtysh -c 'show ip bgp ipv6 summary'; done
      (...)
      =========================
      Thu Oct 12 11:38:26 EDT 2023
      
      IPv4 Unicast Summary (VRF default):
      BGP router identifier 192.168.18.25, local AS number 65101 vrf-id 0
      BGP table version 17264
      RIB entries 199, using 36 KiB of memory
      Peers 1, using 723 KiB of memory
      
      Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
      192.168.18.21   4      65200    660057    657663        0    0    0 00:18:21          100      100 N/A
      
      Total number of neighbors 1
      
      IPv6 Unicast Summary (VRF default):
      BGP router identifier 192.168.18.25, local AS number 65101 vrf-id 0
      BGP table version 9248
      RIB entries 199, using 36 KiB of memory
      Peers 1, using 723 KiB of memory
      
      Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
      192.168.18.21   4      65200    660057    657663        0    0    0 00:18:21          100      100 N/A
      
      Total number of neighbors 1
      =========================
      Thu Oct 12 11:38:28 EDT 2023
      
      IPv4 Unicast Summary (VRF default):
      BGP router identifier 192.168.18.25, local AS number 65101 vrf-id 0
      BGP table version 17364
      RIB entries 4, using 736 bytes of memory
      Peers 1, using 723 KiB of memory
      
      Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
      192.168.18.21   4      65200    660060    657664        0    0    0 00:00:01         Idle        0 N/A
      
      Total number of neighbors 1
      
      IPv6 Unicast Summary (VRF default):
      BGP router identifier 192.168.18.25, local AS number 65101 vrf-id 0
      BGP table version 9348
      RIB entries 1, using 184 bytes of memory
      Peers 1, using 723 KiB of memory
      
      Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
      192.168.18.21   4      65200    660060    657664        0    0    0 00:00:01         Idle        0 N/A
      (...)
      =========================
      Thu Oct 12 11:39:07 EDT 2023
      
      IPv4 Unicast Summary (VRF default):
      BGP router identifier 192.168.18.25, local AS number 65101 vrf-id 0
      BGP table version 17364
      RIB entries 4, using 736 bytes of memory
      Peers 1, using 723 KiB of memory
      
      Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
      192.168.18.21   4      65200    660060    657664        0    0    0 00:00:40       Active        0 N/A
      
      Total number of neighbors 1
      
      IPv6 Unicast Summary (VRF default):
      BGP router identifier 192.168.18.25, local AS number 65101 vrf-id 0
      BGP table version 9348
      RIB entries 1, using 184 bytes of memory
      Peers 1, using 723 KiB of memory
      
      Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
      192.168.18.21   4      65200    660060    657664        0    0    0 00:00:40       Active        0 N/A
      
      Total number of neighbors 1
      =========================
      Thu Oct 12 11:39:09 EDT 2023
      
      IPv4 Unicast Summary (VRF default):
      BGP router identifier 192.168.18.25, local AS number 65101 vrf-id 0
      BGP table version 17364
      RIB entries 4, using 736 bytes of memory
      Peers 1, using 723 KiB of memory
      
      Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
      192.168.18.21   4      65200    660062    657666        0    0    0 00:00:01            0        0 N/A
      
      Total number of neighbors 1
      
      IPv6 Unicast Summary (VRF default):
      BGP router identifier 192.168.18.25, local AS number 65101 vrf-id 0
      BGP table version 9348
      RIB entries 1, using 184 bytes of memory
      Peers 1, using 723 KiB of memory
      
      Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
      192.168.18.21   4      65200    660062    657666        0    0    0 00:00:01            0        0 N/A
      
      Total number of neighbors 1
      =========================
      Thu Oct 12 11:39:11 EDT 2023
      
      IPv4 Unicast Summary (VRF default):
      BGP router identifier 192.168.18.25, local AS number 65101 vrf-id 0
      BGP table version 17454
      RIB entries 179, using 32 KiB of memory
      Peers 1, using 723 KiB of memory
      
      Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
      192.168.18.21   4      65200    660066    657670        0    0    0 00:00:03           90       90 N/A
      
      Total number of neighbors 1
      
      IPv6 Unicast Summary (VRF default):
      BGP router identifier 192.168.18.25, local AS number 65101 vrf-id 0
      BGP table version 9438
      RIB entries 179, using 32 KiB of memory
      Peers 1, using 723 KiB of memory
      
      Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
      192.168.18.21   4      65200    660066    657670        0    0    0 00:00:03           90       90 N/A
      
      Total number of neighbors 1
      

              fpaoline@redhat.com Federico Paolinelli
              akaris@redhat.com Andreas Karis
              Arti Sood Arti Sood
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: