Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-6571

Parallelized implementation of Debezium Engine

XMLWordPrintable

    • Icon: Enhancement Enhancement
    • Resolution: Duplicate
    • Icon: Major Major
    • None
    • None
    • embedded-engine
    • None
    • False
    • None
    • False

      Current embedded engine architecture mimics Kafka Connect architecture/behaviour.
      A batch of Struct messages are received from connector and then serially run through transformations, serialized and delivered.
      The transfromation and serialization can be CPU consuming task.
      The parallelized engine should have a pool to which the Struct is passed and transformation and serialization is done on many messages in parallel. The messages are then delivered serially to the sink in order.

      The follow-up tasks will be

      • delivering messages to sinks in parallel when supported
      • extending connector architecture in a way the incoming data will be decoded and processed in parallel when possible
        • pilot could be PostgreSQL connector - a single thread will read data from replication slot and chunk of bytes representing messages will be processed in parallel

              vjuranek@redhat.com Vojtech Juranek
              jpechane Jiri Pechanec
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: