Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-3883

Hosted ovnkubernetes pods are not being spread among workers evenly

XMLWordPrintable

    • Important
    • None
    • SDN Sprint 228
    • 1
    • Rejected
    • False
    • Hide

      None

      Show
      None

      While doing a PerfScale test of we noticed that the ovnkube pods are not being spread out evenly among the available workers. Instead they are all stacking on a few until they fill up the available allocatable ebs volumes (25 in the case of m5 instances that we see here).

      An example from partway through our 80 hosted cluster test when there were ~30 hosted clusters created/in progress

      There are 24 workers available:

      ```

      $ for i in `oc get nodes l node-role.kubernetes.io/worker=,node-role.kubernetes.io/infra!=,node-role.kubernetes.io/workload!= | egrep -v "NAME" | awk '{ print $1 }'`;    do  echo $i `oc describe node $i | grep -v openshift | grep ovnkube -c`; done
      ip-10-0-129-227.us-west-2.compute.internal 0
      ip-10-0-136-22.us-west-2.compute.internal 25
      ip-10-0-136-29.us-west-2.compute.internal 0
      ip-10-0-147-248.us-west-2.compute.internal 0
      ip-10-0-150-147.us-west-2.compute.internal 0
      ip-10-0-154-207.us-west-2.compute.internal 0
      ip-10-0-156-0.us-west-2.compute.internal 0
      ip-10-0-157-1.us-west-2.compute.internal 4
      ip-10-0-160-253.us-west-2.compute.internal 0
      ip-10-0-161-30.us-west-2.compute.internal 0
      ip-10-0-164-98.us-west-2.compute.internal 0
      ip-10-0-168-245.us-west-2.compute.internal 0
      ip-10-0-170-103.us-west-2.compute.internal 0
      ip-10-0-188-169.us-west-2.compute.internal 25
      ip-10-0-188-194.us-west-2.compute.internal 0
      ip-10-0-191-51.us-west-2.compute.internal 5
      ip-10-0-192-10.us-west-2.compute.internal 0
      ip-10-0-193-200.us-west-2.compute.internal 0
      ip-10-0-193-27.us-west-2.compute.internal 7
      ip-10-0-199-1.us-west-2.compute.internal 0
      ip-10-0-203-161.us-west-2.compute.internal 0
      ip-10-0-204-40.us-west-2.compute.internal 23
      ip-10-0-220-164.us-west-2.compute.internal 0
      ip-10-0-222-59.us-west-2.compute.internal 0

      ```

      This is running quay.io/openshift-release-dev/ocp-release:4.11.11-x86_64 for the hosted clusters and the hypershift operator is quay.io/hypershift/hypershift-operator:4.11 on a 4.11.9 management cluster

              pdiak@redhat.com Patryk Diak
              rhn-support-rzaleski Russell Zaleski
              Russell Zaleski Russell Zaleski
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

                Created:
                Updated:
                Resolved: