Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-859

Schema_only snapshot on idle server - offsets not stored after snapshot

    Details

    • Steps to Reproduce:
      Hide

      create new mysql connector.
      start schema_only snapshot
      make sure there are no changes in mysql server and binlog position remains idle.

      Expected: after snapshot competes offset is stored (logs even says it do)
      Real: offset topic doesn't contain any offset records

      Show
      create new mysql connector. start schema_only snapshot make sure there are no changes in mysql server and binlog position remains idle. Expected: after snapshot competes offset is stored (logs even says it do) Real: offset topic doesn't contain any offset records

      Description

      Mysql connector with snapshot.mode=schema_only. When the source mysql server is idle debezium performes schema snapshot, but doesn't store binlog offset as this happens only when data is produced. For idle server there's no data changes = no offset.

      Now after debezium instance restart or rebalance event the restarted task state machine goes into wrong path thinking it's new job and starts to perform another schema snapshot. This leads to ever growing history topic.

      Some ideas:
      Send one heartbeat when snapshot completes - this will trigger offset storage.
      Send offset periodically by time trigger.

        Gliffy Diagrams

          Attachments

            Activity

              People

              • Assignee:
                jpechanec Jiri Pechanec
                Reporter:
                pimpelsang Eero Koplimets
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: