Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-859

Schema_only snapshot on idle server - offsets not stored after snapshot

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Major
    • 0.9.0.Alpha2
    • 0.8.1.Final
    • mysql-connector
    • None
    • Hide

      create new mysql connector.
      start schema_only snapshot
      make sure there are no changes in mysql server and binlog position remains idle.

      Expected: after snapshot competes offset is stored (logs even says it do)
      Real: offset topic doesn't contain any offset records

      Show
      create new mysql connector. start schema_only snapshot make sure there are no changes in mysql server and binlog position remains idle. Expected: after snapshot competes offset is stored (logs even says it do) Real: offset topic doesn't contain any offset records

    Description

      Mysql connector with snapshot.mode=schema_only. When the source mysql server is idle debezium performes schema snapshot, but doesn't store binlog offset as this happens only when data is produced. For idle server there's no data changes = no offset.

      Now after debezium instance restart or rebalance event the restarted task state machine goes into wrong path thinking it's new job and starts to perform another schema snapshot. This leads to ever growing history topic.

      Some ideas:
      Send one heartbeat when snapshot completes - this will trigger offset storage.
      Send offset periodically by time trigger.

      Attachments

        Activity

          People

            jpechane Jiri Pechanec
            pimpelsang Eero Koplimets (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: