Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-1420

Offset should be saved after snapshot

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Obsolete
    • Icon: Major Major
    • None
    • 0.9.5.Final
    • embedded-engine
    • None
    • Hide

      Run an app with the embedded engine and the MySQL Database from the tutorial. Use any class you want to store the offset. When the snapshot is done, kill the app and restart it. You'll see that most of the time it'll redo a snapshot saying it did not find any snapshot.
      I tried to use offset.flush.interval.ms=5000 in my tests but it seems it applies only to changes and not the snapshot itself).

      Show
      Run an app with the embedded engine and the MySQL Database from the tutorial. Use any class you want to store the offset. When the snapshot is done, kill the app and restart it. You'll see that most of the time it'll redo a snapshot saying it did not find any snapshot. I tried to use offset.flush.interval.ms=5000 in my tests but it seems it applies only to changes and not the snapshot itself).

      I've tested a debezium embedded app with an instance of and noticed that the offset is not stored when the snapshot is done. As soon as a change is sent, the offset is then saved but not before. And when the engine stops it's not always guaranteed that the offset gets saved.

      I think the offset should definitely be saved at the end of a snapshot otherwise if the app restarts it will redo a snapshot for nothing.

      Here are the logs of my tests before doing any change on the database https://pastebin.com/raw/y3YNGkGe.

      Then when I do a change on the db :

      2019-08-01 16:29:24,711 DEBUG  MySQL|ChangeDataSender|binlog  Received query command: Event{header=EventHeaderV4{timestamp=1564691364000, eventType=QUERY, serverId=223344, headerLength=19, dataLength=58, nextPosition=296, flags=8}, data=QueryEventData{threadId=14, executionTime=0, errorCode=0, database='inventory', sql='BEGIN'}}   [io.debezium.connector.mysql.BinlogReader]
...
2019-08-01 16:29:24,996 INFO   MySQL|ChangeDataSender|binlog  1 records sent during previous 00:01:37.847, last recorded offset: {ts_sec=1564691364, file=mysql-bin.000003, pos=219, row=1, server_id=223344, event=2}   [io.debezium.connector.mysql.BinlogReader]
2019-08-01 16:29:24,996 DEBUG  ||  Embedded engine returned from polling task for records   [io.debezium.embedded.EmbeddedEngine]
2019-08-01 16:29:24,996 DEBUG  ||  Received 1 records from the task   [io.debezium.embedded.EmbeddedEngine]
2019-08-01 16:29:25,013 INFO   MySQL|ChangeDataSender|task  Save offset to dynamo table DbzOffsetTable   [main.kotlin.DynamoOffsetBackingStore]

      where you can see it actually saves the offset (in the last log line)

              Unassigned Unassigned
              cscetbon cyril Scetbon (Inactive)
              Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: