Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-4238

Hosted ovnkubernetes pods are not being spread among workers evenly

XMLWordPrintable

    • Important
    • None
    • SDN Sprint 228, SDN Sprint 229, SDN Sprint 230, SDN Sprint 231
    • 4
    • Rejected
    • False
    • Hide

      None

      Show
      None

      This is a clone of issue OCPBUGS-3883. The following is the description of the original issue:

      While doing a PerfScale test of we noticed that the ovnkube pods are not being spread out evenly among the available workers. Instead they are all stacking on a few until they fill up the available allocatable ebs volumes (25 in the case of m5 instances that we see here).

      An example from partway through our 80 hosted cluster test when there were ~30 hosted clusters created/in progress

      There are 24 workers available:

      ```

      $ for i in `oc get nodes l node-role.kubernetes.io/worker=,node-role.kubernetes.io/infra!=,node-role.kubernetes.io/workload!= | egrep -v "NAME" | awk '{ print $1 }'`;    do  echo $i `oc describe node $i | grep -v openshift | grep ovnkube -c`; done
      ip-10-0-129-227.us-west-2.compute.internal 0
      ip-10-0-136-22.us-west-2.compute.internal 25
      ip-10-0-136-29.us-west-2.compute.internal 0
      ip-10-0-147-248.us-west-2.compute.internal 0
      ip-10-0-150-147.us-west-2.compute.internal 0
      ip-10-0-154-207.us-west-2.compute.internal 0
      ip-10-0-156-0.us-west-2.compute.internal 0
      ip-10-0-157-1.us-west-2.compute.internal 4
      ip-10-0-160-253.us-west-2.compute.internal 0
      ip-10-0-161-30.us-west-2.compute.internal 0
      ip-10-0-164-98.us-west-2.compute.internal 0
      ip-10-0-168-245.us-west-2.compute.internal 0
      ip-10-0-170-103.us-west-2.compute.internal 0
      ip-10-0-188-169.us-west-2.compute.internal 25
      ip-10-0-188-194.us-west-2.compute.internal 0
      ip-10-0-191-51.us-west-2.compute.internal 5
      ip-10-0-192-10.us-west-2.compute.internal 0
      ip-10-0-193-200.us-west-2.compute.internal 0
      ip-10-0-193-27.us-west-2.compute.internal 7
      ip-10-0-199-1.us-west-2.compute.internal 0
      ip-10-0-203-161.us-west-2.compute.internal 0
      ip-10-0-204-40.us-west-2.compute.internal 23
      ip-10-0-220-164.us-west-2.compute.internal 0
      ip-10-0-222-59.us-west-2.compute.internal 0

      ```

      This is running quay.io/openshift-release-dev/ocp-release:4.11.11-x86_64 for the hosted clusters and the hypershift operator is quay.io/hypershift/hypershift-operator:4.11 on a 4.11.9 management cluster

            pdiak@redhat.com Patryk Diak
            openshift-crt-jira-prow OpenShift Prow Bot
            Anurag Saxena Anurag Saxena
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: