Uploaded image for project: 'Managed Service - Streams'
  1. Managed Service - Streams
  2. MGDSTRM-8420

determine a better thread num configs in kafka broker

XMLWordPrintable

    • Icon: Spike Spike
    • Resolution: Won't Do
    • Icon: Normal Normal
    • None
    • None
    • None
    • None
    • False
    • None
    • False
    • No
    • ---
    • ---

      WHAT

      Kafka broker has many thread number related configs, that is:

      • num.network.threads -> default to 3
      • num.io.threads -> default to 8
      • num.replica.fetchers -> default to 1 (split out in  MGDSTRM-9196)
      • num.recovery.threads.per.data.dir -> default to 1 (split out in MGDSTRM-9195)
      • log.cleaner.threads -> default to 1
      • background.threads -> default to 10

       

      It looks like the RHOSAK doesn't override any of these configs. But some default values for them are too conservative, ex: num.recovery.threads.per.data.dir is default to 1, which will only have one thread to do log recovery, and let other CPUs idle. Other example is like num.replica.fetchers is also default to 1, but if we increase it, it'll improve the throughput of the follower replication.

      WHY

      Improve the performance 

      HOW

      Run benchmark tests to find out the best configuration for RHOSAK

       

            Unassigned Unassigned
            lukchen@redhat.com Luke Chen
            Kafka Integrations
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: