Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-51971

etcd compaction can become blocking when it shouldn't in 3.5.16+

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Undefined Undefined
    • None
    • 4.17.z, 4.18.z, 4.19.0
    • Etcd
    • None
    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • None
    • None
    • None
    • Proposed
    • Bug Fix
    • A change that went into etcd 3.5.16 can lead to etcd compaction becoming blocking whenever it takes more than 10ms to process a batch. We are correcting that regression in this update.
    • None
    • None
    • None
    • None

      Whenever etcd compaction batches take more than 10ms to perform compaction becomes request blocking. This upstream PR provides more detail

      https://github.com/etcd-io/etcd/pull/19405

      While investigating the kube apiserver request latency spike in 4.19 we have determined that RHEL 9.6 based host OS seems to yield a 10-20x increase in compaction duration. With this PR the impact on apiserver request latency is reduced by approximately 50% yet compaction duration remains high. We're trying to chase that with the kernel team independently.

      Given that 4.17 and later are on 3.5.16+ it would be good to backport this fix through 4.17.

              dwest@redhat.com Dean West
              rhn-support-sdodson Scott Dodson
              None
              None
              Ge Liu Ge Liu
              None
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: