Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-54377

Exporting Router advertisement to external frr taking 5 minutes during scale testing

XMLWordPrintable

    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • None
    • None
    • None
    • Rejected
    • CORENET Sprint 271, CNF Network Sprint 273, CNF Network Sprint 274
    • 3
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Description of problem:

      I am scale testing BGP router export scenario on a OCP 4.19 baremetal cluster with 24 worker nodes (no infra nodes) using 4.19.0-ec.3 with  OVNK BGP image which is built using the PR  build 4.19,openshift/ovn-kubernetes#2239 on 03/25/2025.

      Test creates 72 CUDN, 1 namespace per CUDN and 1 pod per namespace. I waited for 30 minutes after creating all the resources and before creating router advertisements.

      Then test creates 72 router advertisements where each RA advertises only one unique cudn i.e RA:CUDN is 1:1.

      We are monitoring on the baremetal host where the external frr is running for the route updates using netlink sockets. Latency of Router advertisement is the time elapsed from the creation of the RA CRD to the detection of the corresponding route in this external frr host.

      Latency of exporting RA to external frr linearly increasing with the number of RAs and taking more than 5 minutes for 70th RA. 

      Another observation is it is hardly taking 2 seconds for the ping test from this external host (once it detects the route) to the pod inside the cudn.

       

       

      RA Name total time in minutes taken from RA creation to its detection on external frr
      ra-2 0.11
      ra-5 0.16
      ra-0 0.16
      ra-7 0.24
      ra-9 0.41
      ra-4 0.45
      ra-3 0.49
      ra-1 0.49
      ra-13 0.63
      ra-6 0.65
      ra-10 0.74
      ra-16 0.93
      ra-18 0.96
      ra-11 0.97
      ra-17 0.97
      ra-14 0.97
      ra-19 0.99
      ra-27 1.36
      ra-21 1.36
      ra-22 1.47
      ra-29 1.47
      ra-12 1.48
      ra-8 1.48
      ra-15 1.48
      ra-31 1.55
      ra-30 1.55
      ra-32 1.61
      ra-35 1.9
      ra-33 1.9
      ra-34 1.9
      ra-37 1.9
      ra-36 1.94
      ra-25 1.96
      ra-28 1.96
      ra-24 1.96
      ra-26 1.96
      ra-23 1.96
      ra-20 1.98
      ra-38 2.1
      ra-41 2.16
      ra-39 2.27
      ra-42 2.3
      ra-40 2.32
      ra-43 2.4
      ra-45 2.5
      ra-44 2.5
      ra-46 2.7
      ra-47 2.75
      ra-48 2.75
      ra-49 2.88
      ra-50 2.98
      ra-52 3.16
      ra-51 3.21
      ra-53 3.21
      ra-54 3.43
      ra-55 3.49
      ra-56 3.56
      ra-58 3.66
      ra-57 3.67
      ra-59 3.86
      ra-60 3.96
      ra-61 4.06
      ra-62 4.12
      ra-63 4.24
      ra-64 4.34
      ra-65 4.52
      ra-66 4.57
      ra-67 4.67
      ra-68 4.81
      ra-69 4.83
      ra-70 5.19
      ra-71 5.23

       

       

      Test results are avaialble at https://docs.google.com/spreadsheets/d/1WLuTpcrTwFBUcZ-XF2wppOJL9M13_V_HYh2i23UpQUo/edit?usp=sharing 
      Grafana screenshots at https://storage.scalelab.redhat.com/anilvenkata/bgp/ra72export/ 

      I can provide the live environment to the engineer for troubleshooting.

       

       

              rhn-support-arghosh Arnab Ghosh
              vkommadi@redhat.com VENKATA ANIL kumar KOMMADDI
              None
              None
              VENKATA ANIL kumar KOMMADDI VENKATA ANIL kumar KOMMADDI
              None
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

                Created:
                Updated: