Uploaded image for project: 'OpenShift Hosted Control Plane'
  1. OpenShift Hosted Control Plane
  2. HOSTEDCP-626

m6 instance types have incorrect volume limit set

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Blocker Blocker
    • openshift 4.12
    • openshift-4.11.z
    • None
    • False
    • None
    • False
    • Hide

      Install OSD/OpenShift cluster with m6i or m6a as the workers for 3 worker nodes.

      Run oc describe node on a worker to see attachable-volumes-aws-ebs set to 39.

      To reproduce the failure of ovnkube to launch you will also need to install a max amount of Hosted Clusters (9 if using 3 workers). Depending on where they are scheduled you may or may not run into this issue as if they are evenly spread out it will not error.

      Show
      Install OSD/OpenShift cluster with m6i or m6a as the workers for 3 worker nodes. Run oc describe node on a worker to see attachable-volumes-aws-ebs set to 39. To reproduce the failure of ovnkube to launch you will also need to install a max amount of Hosted Clusters (9 if using 3 workers). Depending on where they are scheduled you may or may not run into this issue as if they are evenly spread out it will not error.
    • Hypershift Sprint 20
    • Critical
    • 10
    • 0
    • 0

      During PerfScale testing of 80 Hosted Clusters on 24 m6i workers we ran into an issue where OpenShift reports the max allocate-able EBS volumes to be 39 (via oc describe node) however anything above 27 attached EBS volumes causes the attachment to never complete and sit with a status of "attaching."

       

      NOTE: both m6i and m6a instance types reported this as 39

       

      On m5 instance types this number is set to 25 and thus does not run into this issue. Ultimately, this caused many ovnkube pods to be unable to start as their pvs were not present and they timed out.

       

      Example pod output:
      Warning FailedAttachVolume 96s (x27 over 40m) attachdetach-controller AttachVolume.Attach failed for volume "pvc-763be113-1923-4462-92b5-62f64517578d" : rpc error: code = Internal desc = Could not attach volume "vol-03441aff5f07550bd" to node "i-09190d2d740bf2212": attachment of disk "vol-03441aff5f07550bd" failed, expected device to be attached but was attaching

              sjenning Seth Jennings
              rhn-support-rzaleski Russell Zaleski
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: