Uploaded image for project: 'AMQ Streams'
  1. AMQ Streams
  2. ENTMQST-3893

[KAFKA] Investigating performance impacts of Kafka client heap ByteBuffer pooling

XMLWordPrintable

    • Icon: Task Task
    • Resolution: Unresolved
    • Icon: Major Major
    • None
    • None
    • None
    • False
    • None
    • False

      Both BufferPool and MemoryPool (former used on producer side, latter on Selector side) uses heap-based ByteBuffer by default, but Java NIO channels must perform an additional copy into a direct ByteBuffer to interact to the socket(s), see jdk 11 IOUtil for reference.

      This will both impact producers and consumers, with the former mostly affected depending the amount of batching performed: it worths investigate with appropriate profiling tools and a test (very likely the OpenMessaging benchmark) the impact while switching to just use direct ByteBuffer(s) in such pools.

      The additional benefit of the change is to increase predictable resource usage/release: an abandoned producer/connection keeps around the thread local direct ByteBuffer pool allocated by IOUtil until GC would decide to get rid of it, while interacting directly with a direct buffer/memory pool enable deterministic resource release, although it requires interacting with the Cleaner API (not officially supported on Java 8).
      In relation to Cleaners, Netty approach seems the right way to preserve compatibility across JDKs versions: Netty 4.1 Cleaner.

            Unassigned Unassigned
            fnigro Francesco Nigro
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: