Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-390 Data are read from the binlog and not written into kafka
  3. DBZ-405

Modify BinlogReader to process transactions via buffer

XMLWordPrintable

    • Icon: Sub-task Sub-task
    • Resolution: Done
    • Icon: Major Major
    • 0.7.0
    • 0.6
    • mysql-connector
    • None

      DBZ-390 has shown that the replication log can contain rolled-back transactions. We cannot stream the transaction directly into the Kafka but we need to buffer events before BEGIN and COMMIT. Only if transaction is completed with COMMIT we will stream it to Kafka from the buffer.
      We will store the events temporarily into a bounded buffer with a configurable capacity. When the transaction is committed we will play it from the buffer.
      It is necessary to be careful with offset commits to make sure that the recovery is reliable and no data are lost when in memory state is lost on a connector restart.

              jpechane Jiri Pechanec
              jpechane Jiri Pechanec
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: