Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-859

Schema_only snapshot on idle server - offsets not stored after snapshot

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • 0.9.0.Alpha2
    • 0.8.1.Final
    • mysql-connector
    • None
    • Hide

      create new mysql connector.
      start schema_only snapshot
      make sure there are no changes in mysql server and binlog position remains idle.

      Expected: after snapshot competes offset is stored (logs even says it do)
      Real: offset topic doesn't contain any offset records

      Show
      create new mysql connector. start schema_only snapshot make sure there are no changes in mysql server and binlog position remains idle. Expected: after snapshot competes offset is stored (logs even says it do) Real: offset topic doesn't contain any offset records

      Mysql connector with snapshot.mode=schema_only. When the source mysql server is idle debezium performes schema snapshot, but doesn't store binlog offset as this happens only when data is produced. For idle server there's no data changes = no offset.

      Now after debezium instance restart or rebalance event the restarted task state machine goes into wrong path thinking it's new job and starts to perform another schema snapshot. This leads to ever growing history topic.

      Some ideas:
      Send one heartbeat when snapshot completes - this will trigger offset storage.
      Send offset periodically by time trigger.

              jpechane Jiri Pechanec
              pimpelsang Eero Koplimets (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: