Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-4238

Hosted ovnkubernetes pods are not being spread among workers evenly

    XMLWordPrintable

Details

    • Important
    • SDN Sprint 228, SDN Sprint 229, SDN Sprint 230, SDN Sprint 231
    • 4
    • Rejected
    • False
    • Hide

      None

      Show
      None

    Description

      This is a clone of issue OCPBUGS-3883. The following is the description of the original issue:

      While doing a PerfScale test of we noticed that the ovnkube pods are not being spread out evenly among the available workers. Instead they are all stacking on a few until they fill up the available allocatable ebs volumes (25 in the case of m5 instances that we see here).

      An example from partway through our 80 hosted cluster test when there were ~30 hosted clusters created/in progress

      There are 24 workers available:

      ```

      $ for i in `oc get nodes l node-role.kubernetes.io/worker=,node-role.kubernetes.io/infra!=,node-role.kubernetes.io/workload!= | egrep -v "NAME" | awk '{ print $1 }'`;    do  echo $i `oc describe node $i | grep -v openshift | grep ovnkube -c`; done
      ip-10-0-129-227.us-west-2.compute.internal 0
      ip-10-0-136-22.us-west-2.compute.internal 25
      ip-10-0-136-29.us-west-2.compute.internal 0
      ip-10-0-147-248.us-west-2.compute.internal 0
      ip-10-0-150-147.us-west-2.compute.internal 0
      ip-10-0-154-207.us-west-2.compute.internal 0
      ip-10-0-156-0.us-west-2.compute.internal 0
      ip-10-0-157-1.us-west-2.compute.internal 4
      ip-10-0-160-253.us-west-2.compute.internal 0
      ip-10-0-161-30.us-west-2.compute.internal 0
      ip-10-0-164-98.us-west-2.compute.internal 0
      ip-10-0-168-245.us-west-2.compute.internal 0
      ip-10-0-170-103.us-west-2.compute.internal 0
      ip-10-0-188-169.us-west-2.compute.internal 25
      ip-10-0-188-194.us-west-2.compute.internal 0
      ip-10-0-191-51.us-west-2.compute.internal 5
      ip-10-0-192-10.us-west-2.compute.internal 0
      ip-10-0-193-200.us-west-2.compute.internal 0
      ip-10-0-193-27.us-west-2.compute.internal 7
      ip-10-0-199-1.us-west-2.compute.internal 0
      ip-10-0-203-161.us-west-2.compute.internal 0
      ip-10-0-204-40.us-west-2.compute.internal 23
      ip-10-0-220-164.us-west-2.compute.internal 0
      ip-10-0-222-59.us-west-2.compute.internal 0

      ```

      This is running quay.io/openshift-release-dev/ocp-release:4.11.11-x86_64 for the hosted clusters and the hypershift operator is quay.io/hypershift/hypershift-operator:4.11 on a 4.11.9 management cluster

      Attachments

        Issue Links

          Activity

            People

              pdiak@redhat.com Patryk Diak
              openshift-crt-jira-prow OpenShift Prow Bot
              Anurag Saxena Anurag Saxena
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: