Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-19665

KubeAPI is unable to automatically compact/defrag the etcd DB if it goes over the 8GB size limit

XMLWordPrintable

    • No
    • False
    • Hide

      None

      Show
      None

      Description of problem:

      If there's a workload which creates or rotates through a lot of Kubernetes objects (like namespaces) the etcd DB size can easily reach the 8GB limit in 5min between the Kubernetes API automated defrags.
      
      When the limit is reached etcd will arm alarm:NOSPACE which prevents any future attempts by Kubernetes API to compact and defrag the database.

      Version-Release number of selected component (if applicable):

       

      How reproducible:

      Run a workload which creates and deletes namespaces quickly.

      Steps to Reproduce:

      1.Create and delete Kubernetes objects until etcd DB reaches more than 8GB within 5min
      2.Etcd will lock up
      3.
      

      Actual results:

      Etcd arms alarm:nospace, this requires a manual intervention to compact and defrag.

      Expected results:

      Etcd automatically compacts and defrags, possibly sends an alert to investigate the issue further.

      Additional info:

       

            dwest@redhat.com Dean West
            ljakubow2.openshift Leszek Jakubowski
            ge liu ge liu
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: