Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-1420

Offset should be saved after snapshot

    XMLWordPrintable

Details

    • Bug
    • Resolution: Obsolete
    • Major
    • None
    • 0.9.5.Final
    • embedded-engine
    • None
    • Hide

      Run an app with the embedded engine and the MySQL Database from the tutorial. Use any class you want to store the offset. When the snapshot is done, kill the app and restart it. You'll see that most of the time it'll redo a snapshot saying it did not find any snapshot.
      I tried to use offset.flush.interval.ms=5000 in my tests but it seems it applies only to changes and not the snapshot itself).

      Show
      Run an app with the embedded engine and the MySQL Database from the tutorial. Use any class you want to store the offset. When the snapshot is done, kill the app and restart it. You'll see that most of the time it'll redo a snapshot saying it did not find any snapshot. I tried to use offset.flush.interval.ms=5000 in my tests but it seems it applies only to changes and not the snapshot itself).

    Description

      I've tested a debezium embedded app with an instance of and noticed that the offset is not stored when the snapshot is done. As soon as a change is sent, the offset is then saved but not before. And when the engine stops it's not always guaranteed that the offset gets saved.

      I think the offset should definitely be saved at the end of a snapshot otherwise if the app restarts it will redo a snapshot for nothing.

      Here are the logs of my tests before doing any change on the database https://pastebin.com/raw/y3YNGkGe.

      Then when I do a change on the db :

      2019-08-01 16:29:24,711 DEBUG  MySQL|ChangeDataSender|binlog  Received query command: Event{header=EventHeaderV4{timestamp=1564691364000, eventType=QUERY, serverId=223344, headerLength=19, dataLength=58, nextPosition=296, flags=8}, data=QueryEventData{threadId=14, executionTime=0, errorCode=0, database='inventory', sql='BEGIN'}}   [io.debezium.connector.mysql.BinlogReader]
...
2019-08-01 16:29:24,996 INFO   MySQL|ChangeDataSender|binlog  1 records sent during previous 00:01:37.847, last recorded offset: {ts_sec=1564691364, file=mysql-bin.000003, pos=219, row=1, server_id=223344, event=2}   [io.debezium.connector.mysql.BinlogReader]
2019-08-01 16:29:24,996 DEBUG  ||  Embedded engine returned from polling task for records   [io.debezium.embedded.EmbeddedEngine]
2019-08-01 16:29:24,996 DEBUG  ||  Received 1 records from the task   [io.debezium.embedded.EmbeddedEngine]
2019-08-01 16:29:25,013 INFO   MySQL|ChangeDataSender|task  Save offset to dynamo table DbzOffsetTable   [main.kotlin.DynamoOffsetBackingStore]

      where you can see it actually saves the offset (in the last log line)

      Attachments

        Issue Links

          is incorporated by

          Bug - A problem that impairs or prevents the functions of the product. DBZ-5464 Snapshot result not saved if LAST record is filtered out

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed

          Activity

            People

              Unassigned Unassigned
              cscetbon cyril Scetbon (Inactive)
              Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: