Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-165

Spike in pod-latency graph observed due to ovnkube-master restarts

    XMLWordPrintable

Details

    • Important
    • False
    • Hide

      None

      Show
      None

    Description

      Description of problem:

      When running node-density (245 pods/node) on a 120 node cluster, we see that there is a huge spike (~22s) in Avg pod-latency. When the spike occurs we see all the ovnkube-master pods go through a restart. 

      The restart happens because of (ovnkube-master pods)

      2022-08-10T04:04:44.494945179Z panic: reflect: call of reflect.Value.Len on ptr Value

      Version-Release number of selected component (if applicable):

      4.12.0-0.nightly-2022-08-09-114621

      How reproducible:

      Steps to Reproduce:
      1. Run node-density on a 120 node cluster

      Actual results:

      Spike observed in pod-latency graph ~22s

      Expected results:

      Steady pod-latency graph ~4s

      Additional info:

      Attachments

        Issue Links

          blocks

          Bug - A problem that impairs or prevents the functions of the product. OCPBUGS-1012 Spike in pod-latency graph observed due to ovnkube-master restarts

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed
          is cloned by

          Bug - A problem that impairs or prevents the functions of the product. OCPBUGS-1012 Spike in pod-latency graph observed due to ovnkube-master restarts

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed

          Bug - A problem that impairs or prevents the functions of the product. OCPBUGS-4598 [4.10z] Spike in pod-latency graph observed due to ovnkube-master restarts

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed

          Bug - A problem that impairs or prevents the functions of the product. OCPBUGS-9979 IBM Cloud: Node density tests fail due to increased pod latency times

          • Normal - To be worked after urgent and high priorities are resolved. This term is chosen when the severity of an issue is relatively close to the level of effort to fix it. The existence of an easily implemented workaround can also lead to this priority level instead of a higher priority.
          • Closed
          is depended on by

          Story - Used to indicate the smallest unit of work for a team to complete and is end user-facing. Created by Jira Software - do not edit or delete. SDN-3079 Ensure 250 node scale tests match the required SLOs

          • Critical - To be worked on immediately following BLOCKER issues.
          • To Do
          links to

          GitHub openshift/ovn-kubernetes#1255: OCPBUGS-165: [DownstreamMerge] 8-29-2022 merge

          Web Link upstream ovnk fix

          Activity

            People

              trozet@redhat.com Tim Rozet
              mohit-sheth Mohit Jitendra Sheth
              Michael Fiedler Michael Fiedler
              Votes:
              1 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: