Uploaded image for project: 'MicroShift'
  1. MicroShift
  2. USHIFT-5793

MicroShift 4.19 pod ready node density perf regression

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Not a Bug
    • Icon: Normal Normal
    • None
    • openshift-4.19
    • Core
    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      As part of the Node Density tests that we do on MicroShift to ensure consistent performance, we measure the time that it takes for the pod to become in ready state. 

       

      Normally we see increasing pod ready latencies as we add additional conditions for their readiness (such as network policies). Additionally as we increase the queries per second (QPS) or the rate of pod creation we see increases in pod ready latency.

      While testing 4.19-rc2 we see pod ready latencies nearly 3x higher than seen before. Normally we see the 99th percentile around 4-5s and that has increased to 13-15s. All measured metrics around pod ready including mean, and max are proportionally increased as well.

      It is fully reproducible not only between clean installations, but also between various node-density variations (adding different conditions/readiness probes to pod creation).

      I've attached a chart demonstrating the pod ready increase between the last few versions. 

      I've seen a similar and potentially related bug in https://issues.redhat.com/browse/OCPBUGS-50522.

              Unassigned Unassigned
              sejug2 Sebastian Jug
              None
              None
              None
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: