Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-18389

4.14 after ec4 has a higher pod ready latency compared to 4.13.10

XMLWordPrintable

    • Important
    • No
    • SDN Sprint 241, SDN Sprint 242
    • 2
    • Approved
    • False
    • Hide

      None

      Show
      None

      Description of problem:

      4.14.0-0.nightly-2023-08-28-154013 has a higher pod ready latency(3s +) compared to 4.14 ec4 and 4.13.10(~1s) in node-density (lite) test

      Version-Release number of selected component (if applicable):

      4.14.0-0.nightly-2023-08-28-154013

      How reproducible:

      every time

      Steps to Reproduce:

      1. Install a cluster and scale up to 24 worker nodes, install 3 infra nodes and move monitoring, ingress, registry components to infra nodes. 
I used vsphere-8 on IBM Cloud. 
Mastre and worker node type: 8 vCPU, 16G
Infra node size: 48 vCPU, 196G
2. Run node-density (lite) test with 245 pod per node
3. Compare the pod ready latency to 4.13.z, and 4.14 ec4

      Actual results:

      4.14.0-0.nightly-2023-08-28-154013 has a higher pod ready latency compared to 4.14 ec4 and 4.13.10

      Expected results:

      4.14 should have similar pod ready latency compared to previous release

      Additional info:

       
      OCP Version Flexy Id Scale Ci Job Grafana URL Cloud Arch Type Network Type Worker Count PODS_PER_NODE Avg Pod Ready (ms) P99 Pod Ready (ms) 
      4.14.0-0.nightly-2023-08-28-154013 229395 198 dac00d0d-ab0e-4b35-bd08-55c856edc742 vsphere amd64 OVN 24 245 3153 11991
      4.14.0-0.nightly-2023-08-28-154013 test2 229720 220 b81ac8e3-77ef-4e00-8e4c-3c23a4d2b2b6 vsphere amd64 OVN 24 245 3527 12483
      4.13.10 229585 212 c4bd2ad5-4e1a-48b7-ba98-180098c52c58 vsphere amd64 OVN 24 245 1045 1973
      4.13.10 test2 229585 217 64d2df2f-26e3-4060-8276-7271234a8620 vsphere amd64 OVN 24 245 1047 2050
      4.14.0-ec.4 229721 219 c3866a3b-7338-401b-a8cf-97db49d04d62 vsphere amd64 OVN 24 245 962 2066

       

      Grafana dashboard compare link:
      https://grafana.rdu2.scalelab.redhat.com:3000/d/g4dJlkBnz3/kube-burner-compare?orgId=1&var-Datasource=QE%20kube-burner&var-platform=VSphere&var-sdn=OVNKubernetes&var-workload=node-density&var-nodes=31&var-uuid=64d2df2f-26e3-4060-8276-7271234a8620&var-uuid=b81ac8e3-77ef-4e00-8e4c-3c23a4d2b2b6&var-uuid=c3866a3b-7338-401b-a8cf-97db49d04d62&var-uuid=c4bd2ad5-4e1a-48b7-ba98-180098c52c58&var-uuid=dac00d0d-ab0e-4b35-bd08-55c856edc742&var-latencyPercentile=P99&var-condition=Ready&var-component=etcd&var-compare_by=metadata.ocpVersion&var-master_cores=8&var-worker_cores=8

       

      must-gathers are uploaded to: OCPBUGS-18389 - Google Drive

            dosmith Douglas Smith
            rhn-support-qili Qiujie Li
            Anurag Saxena Anurag Saxena
            Surya Seetharaman
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: