Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-390 Data are read from the binlog and not written into kafka
  3. DBZ-405

Modify BinlogReader to process transactions via buffer

    XMLWordPrintable

Details

    • Sub-task
    • Resolution: Done
    • Major
    • 0.7.0
    • 0.6
    • mysql-connector
    • None

    Description

      DBZ-390 has shown that the replication log can contain rolled-back transactions. We cannot stream the transaction directly into the Kafka but we need to buffer events before BEGIN and COMMIT. Only if transaction is completed with COMMIT we will stream it to Kafka from the buffer.
      We will store the events temporarily into a bounded buffer with a configurable capacity. When the transaction is committed we will play it from the buffer.
      It is necessary to be careful with offset commits to make sure that the recovery is reliable and no data are lost when in memory state is lost on a connector restart.

      Attachments

        Issue Links

          Activity

            People

              jpechane Jiri Pechanec
              jpechane Jiri Pechanec
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: