Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-8563

Pulsar Producer Batcher Builder - Key Based Batching

XMLWordPrintable

    • Icon: Feature Request Feature Request
    • Resolution: Done
    • Icon: Major Major
    • 3.1.0.Alpha1
    • None
    • debezium-server
    • None

      I am using Debezium Server version 3.0.4 Final, connected to a PostgreSQL and producing events using Outbox Event Router to Pulsar. In my specific use case, I would like to use the KeyShared subscription with batching in my consumers.

      According to the Pulsar documentation, key-based batching needs to be enabled on the producer side. The producer config should look like this:

      Producer<byte[]> producer = client.newProducer()
              .topic("my-topic")
              .batcherBuilder(BatcherBuilder.KEY_BASED)
              .create(); 

      The documentation also highlights the requirement for enabling KEY_BASED batching for compatibility with KeyShared subscriptions: https://pulsar.apache.org/docs/2.10.x/client-libraries-java/#key_shared

      However, I couldn't find any way to configure the Debezium Pulsar producer to enable key-based batching. Looking at the Debezium codebase: https://github.com/debezium/debezium-server/blob/v3.0.6.Final/debezium-server-pulsar/src/main/java/io/debezium/server/pulsar/PulsarChangeConsumer.java#L111-L122

      It seems that the default batcher builder is being used for the producer. Am I overlooking something? Is there currently a way to enable key-based batching for the Pulsar producer in Debezium server, or is this a limitation?

       

              Unassigned Unassigned
              chseki CHRISTIAN SEKI
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: