XMLWordPrintable

    • Icon: Feature Request Feature Request
    • Resolution: Unresolved
    • Icon: Undefined Undefined
    • None
    • None
    • etcd
    • None
    • Product / Portfolio Work
    • None
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Etcd started growing up suddenly and dramatically in a large production cluster managed by one of our partners. It went from 160K objects until 750K in around 7 hours and the database size reached near 6.5 GiB. As a consequence of that, 2 of the Etcd nodes crashed and put the cluster in risk.

      There are tools available to monitor the size of the database, but the flood happened so fast that it was not detected by the team administrating the cluster. In addition, the highest peak happened in the small hours.

      We have detected partially the root cause of this flood and we are working to stop it. However, the same may happen if something or someone else starts creating Etcd objects without control. There is no limit for that as far as I know.

      Depending on the circumstances, especially how optimal the infrastructure below is; some clusters can manage large Etcd databases. However, this is not the case for most of the users. I also assume that even the most optimized clusters become overloaded when Etcd reaches a certain size and therefore it does not seem to make sense alowing the creation of new API objects when that happens. Therefore, my request is that OpenShift shall block the creation of new Etcd objects when a specific size is reached. The most complex decision may be to determine which size exactly. For that, I think there may be two suitable options (non-exclusive):

      • Hardcoding a conservative value between 4 and 6 GiB.
      • Making the value configurable.

      For reference, it is limited to 4 GiB on IBM Cloud.

              racedoro@redhat.com Ramon Acedo
              rhn-support-llopezmo Lucas López Montero
              None
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated:
                None
                None