Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-2604

Increase ControlPlane disk IOPS on GCP

    XMLWordPrintable

Details

    • Proposed
    • False
    • Hide

      None

      Show
      None

    Description

      Background:

      https://issues.redhat.com/browse/OCPBUGS-1130

      https://github.com/openshift/release/pull/33248

       

      Description of problem:

      In the CI we figured that etcd is having high latencies on WAL appends, this triggered a wide range of alerts. 

We chased this down to slow disks where we suspect the sequential IOPS to not be enough anymore. In order for us to increase IOPS we can increase the size.

      Version-Release number of selected component (if applicable):

      We suspect this to be > 4.8, but with 4.11/4.12 it's the most noticeable:
https://testgrid.k8s.io/redhat-openshift-ocp-release-4.12-informing#periodic-ci-openshift-release-master-ci-4.12-upgrade-from-stable-4.11-e2e-gcp-sdn-upgrade&width=20

      How reproducible:

      About every second of the above jobs are failing due to that.

      Steps to Reproduce:

      1.
2.
3.

      Actual results:

       

      Expected results:

      etcd not tripping over high disk latency anymore.

      Additional info:

       

      Attachments

        Issue Links

          blocks

          Bug - A problem that impairs or prevents the functions of the product. OCPBUGS-1130 Investigate etcdGRPCRequestsSlow test

          • Critical - To be worked on immediately following BLOCKER issues.
          • MODIFIED

          Activity

            People

              Unassigned Unassigned
              tjungblu@redhat.com Thomas Jungblut
              Gaoyun Pei Gaoyun Pei
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: