Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-20501

When updating lots of services, metallb sessions flap between established, active and idle

XMLWordPrintable

    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • No
    • None
    • None
    • None
    • CNF Network Sprint 243, CNF Network Sprint 244, CNF Network Sprint 245
    • 3
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      When updating lots of services, metallb sessions flap between established, active and idle

      See instructions in the folder shared in the private comment.

      You can reproduce this with:

      for f in _output/*; do         oc apply -f $f &       done
      

      and the files that I shared.

      However, the oc apply does not have to run in the background. You can also just run it with:

      for f in _output/*; do         oc apply -f $f ;       done
      

      and the result / impact on metallb and frr will be the same, the sessions flap for a long time.

      This gets particularly bad when deleting a few services - when sequentially deleting a few services, about 1 per second, FRR just goes into "active" state and bgp sessions are down / active for 40 seconds in my test:

      for f in _output/*; do         oc delete -f $f ; sleep 1;       done
      

      Yields on the peers (example output from one of them):

       while true; do sleep 2 ; echo =========================; date; vtysh -c 'show ip bgp summ'; vtysh -c 'show ip bgp ipv6 summary'; done
      (...)
      =========================
      Thu Oct 12 11:38:26 EDT 2023
      
      IPv4 Unicast Summary (VRF default):
      BGP router identifier 192.168.18.25, local AS number 65101 vrf-id 0
      BGP table version 17264
      RIB entries 199, using 36 KiB of memory
      Peers 1, using 723 KiB of memory
      
      Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
      192.168.18.21   4      65200    660057    657663        0    0    0 00:18:21          100      100 N/A
      
      Total number of neighbors 1
      
      IPv6 Unicast Summary (VRF default):
      BGP router identifier 192.168.18.25, local AS number 65101 vrf-id 0
      BGP table version 9248
      RIB entries 199, using 36 KiB of memory
      Peers 1, using 723 KiB of memory
      
      Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
      192.168.18.21   4      65200    660057    657663        0    0    0 00:18:21          100      100 N/A
      
      Total number of neighbors 1
      =========================
      Thu Oct 12 11:38:28 EDT 2023
      
      IPv4 Unicast Summary (VRF default):
      BGP router identifier 192.168.18.25, local AS number 65101 vrf-id 0
      BGP table version 17364
      RIB entries 4, using 736 bytes of memory
      Peers 1, using 723 KiB of memory
      
      Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
      192.168.18.21   4      65200    660060    657664        0    0    0 00:00:01         Idle        0 N/A
      
      Total number of neighbors 1
      
      IPv6 Unicast Summary (VRF default):
      BGP router identifier 192.168.18.25, local AS number 65101 vrf-id 0
      BGP table version 9348
      RIB entries 1, using 184 bytes of memory
      Peers 1, using 723 KiB of memory
      
      Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
      192.168.18.21   4      65200    660060    657664        0    0    0 00:00:01         Idle        0 N/A
      (...)
      =========================
      Thu Oct 12 11:39:07 EDT 2023
      
      IPv4 Unicast Summary (VRF default):
      BGP router identifier 192.168.18.25, local AS number 65101 vrf-id 0
      BGP table version 17364
      RIB entries 4, using 736 bytes of memory
      Peers 1, using 723 KiB of memory
      
      Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
      192.168.18.21   4      65200    660060    657664        0    0    0 00:00:40       Active        0 N/A
      
      Total number of neighbors 1
      
      IPv6 Unicast Summary (VRF default):
      BGP router identifier 192.168.18.25, local AS number 65101 vrf-id 0
      BGP table version 9348
      RIB entries 1, using 184 bytes of memory
      Peers 1, using 723 KiB of memory
      
      Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
      192.168.18.21   4      65200    660060    657664        0    0    0 00:00:40       Active        0 N/A
      
      Total number of neighbors 1
      =========================
      Thu Oct 12 11:39:09 EDT 2023
      
      IPv4 Unicast Summary (VRF default):
      BGP router identifier 192.168.18.25, local AS number 65101 vrf-id 0
      BGP table version 17364
      RIB entries 4, using 736 bytes of memory
      Peers 1, using 723 KiB of memory
      
      Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
      192.168.18.21   4      65200    660062    657666        0    0    0 00:00:01            0        0 N/A
      
      Total number of neighbors 1
      
      IPv6 Unicast Summary (VRF default):
      BGP router identifier 192.168.18.25, local AS number 65101 vrf-id 0
      BGP table version 9348
      RIB entries 1, using 184 bytes of memory
      Peers 1, using 723 KiB of memory
      
      Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
      192.168.18.21   4      65200    660062    657666        0    0    0 00:00:01            0        0 N/A
      
      Total number of neighbors 1
      =========================
      Thu Oct 12 11:39:11 EDT 2023
      
      IPv4 Unicast Summary (VRF default):
      BGP router identifier 192.168.18.25, local AS number 65101 vrf-id 0
      BGP table version 17454
      RIB entries 179, using 32 KiB of memory
      Peers 1, using 723 KiB of memory
      
      Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
      192.168.18.21   4      65200    660066    657670        0    0    0 00:00:03           90       90 N/A
      
      Total number of neighbors 1
      
      IPv6 Unicast Summary (VRF default):
      BGP router identifier 192.168.18.25, local AS number 65101 vrf-id 0
      BGP table version 9438
      RIB entries 179, using 32 KiB of memory
      Peers 1, using 723 KiB of memory
      
      Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
      192.168.18.21   4      65200    660066    657670        0    0    0 00:00:03           90       90 N/A
      
      Total number of neighbors 1
      

              fpaoline@redhat.com Federico Paolinelli
              akaris@redhat.com Andreas Karis
              None
              None
              Arti Sood Arti Sood
              None
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: