Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-1012

Spike in pod-latency graph observed due to ovnkube-master restarts

    XMLWordPrintable

Details

    • Important
    • False
    • Hide

      None

      Show
      None

    Description

      Description of problem:

      When running node-density (245 pods/node) on a 120 node cluster, we see that there is a huge spike (~22s) in Avg pod-latency. When the spike occurs we see all the ovnkube-master pods go through a restart. 

      The restart happens because of (ovnkube-master pods)

      2022-08-10T04:04:44.494945179Z panic: reflect: call of reflect.Value.Len on ptr Value

      Version-Release number of selected component (if applicable):

      4.12.0-0.nightly-2022-08-09-114621

      How reproducible:

      Steps to Reproduce:
      1. Run node-density on a 120 node cluster

      Actual results:

      Spike observed in pod-latency graph ~22s

      Expected results:

      Steady pod-latency graph ~4s

      Additional info:

      Attachments

        Issue Links

          Activity

            People

              trozet@redhat.com Tim Rozet
              mohit-sheth Mohit Jitendra Sheth
              Michael Fiedler Michael Fiedler
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: