Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-6993

Hosted ovnkubernetes pods are not being spread among workers evenly


    • Important
    • SDN Sprint 232
    • 1
    • Rejected
    • False
    • Hide



      This is a clone of issue OCPBUGS-4238. The following is the description of the original issue:

      This is a clone of issue OCPBUGS-3883. The following is the description of the original issue:

      While doing a PerfScale test of we noticed that the ovnkube pods are not being spread out evenly among the available workers. Instead they are all stacking on a few until they fill up the available allocatable ebs volumes (25 in the case of m5 instances that we see here).

      An example from partway through our 80 hosted cluster test when there were ~30 hosted clusters created/in progress

      There are 24 workers available:


      $ for i in `oc get nodes l node-role.kubernetes.io/worker=,node-role.kubernetes.io/infra!=,node-role.kubernetes.io/workload!= | egrep -v "NAME" | awk '{ print $1 }'`;    do  echo $i `oc describe node $i | grep -v openshift | grep ovnkube -c`; done
      ip-10-0-129-227.us-west-2.compute.internal 0
      ip-10-0-136-22.us-west-2.compute.internal 25
      ip-10-0-136-29.us-west-2.compute.internal 0
      ip-10-0-147-248.us-west-2.compute.internal 0
      ip-10-0-150-147.us-west-2.compute.internal 0
      ip-10-0-154-207.us-west-2.compute.internal 0
      ip-10-0-156-0.us-west-2.compute.internal 0
      ip-10-0-157-1.us-west-2.compute.internal 4
      ip-10-0-160-253.us-west-2.compute.internal 0
      ip-10-0-161-30.us-west-2.compute.internal 0
      ip-10-0-164-98.us-west-2.compute.internal 0
      ip-10-0-168-245.us-west-2.compute.internal 0
      ip-10-0-170-103.us-west-2.compute.internal 0
      ip-10-0-188-169.us-west-2.compute.internal 25
      ip-10-0-188-194.us-west-2.compute.internal 0
      ip-10-0-191-51.us-west-2.compute.internal 5
      ip-10-0-192-10.us-west-2.compute.internal 0
      ip-10-0-193-200.us-west-2.compute.internal 0
      ip-10-0-193-27.us-west-2.compute.internal 7
      ip-10-0-199-1.us-west-2.compute.internal 0
      ip-10-0-203-161.us-west-2.compute.internal 0
      ip-10-0-204-40.us-west-2.compute.internal 23
      ip-10-0-220-164.us-west-2.compute.internal 0
      ip-10-0-222-59.us-west-2.compute.internal 0


      This is running quay.io/openshift-release-dev/ocp-release:4.11.11-x86_64 for the hosted clusters and the hypershift operator is quay.io/hypershift/hypershift-operator:4.11 on a 4.11.9 management cluster

            pdiak@redhat.com Patryk Diak
            openshift-crt-jira-prow OpenShift Prow Bot
            Anurag Saxena Anurag Saxena
            0 Vote for this issue
            5 Start watching this issue