Uploaded image for project: 'Managed Service - Streams'
  1. Managed Service - Streams
  2. MGDSTRM-8420

determine a better thread num configs in kafka broker

XMLWordPrintable

    • Icon: Spike Spike
    • Resolution: Won't Do
    • Icon: Normal Normal
    • None
    • None
    • None
    • None
    • False
    • None
    • False
    • No
    • ---
    • ---

      WHAT

      Kafka broker has many thread number related configs, that is:

      • num.network.threads -> default to 3
      • num.io.threads -> default to 8
      • num.replica.fetchers -> default to 1 (split out in  MGDSTRM-9196)
      • num.recovery.threads.per.data.dir -> default to 1 (split out in MGDSTRM-9195)
      • log.cleaner.threads -> default to 1
      • background.threads -> default to 10

       

      It looks like the RHOSAK doesn't override any of these configs. But some default values for them are too conservative, ex: num.recovery.threads.per.data.dir is default to 1, which will only have one thread to do log recovery, and let other CPUs idle. Other example is like num.replica.fetchers is also default to 1, but if we increase it, it'll improve the throughput of the follower replication.

      WHY

      Improve the performance 

      HOW

      Run benchmark tests to find out the best configuration for RHOSAK

       

              Unassigned Unassigned
              lukchen@redhat.com Luke Chen
              Kafka Integrations
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: