Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-7902

Add configurable delay after successful snapshot before starting streaming

XMLWordPrintable

    • Icon: Feature Request Feature Request
    • Resolution: Done
    • Icon: Minor Minor
    • 2.7.0.Beta1
    • None
    • core-library
    • None
    • False
    • None
    • False

      It is observed that immediately after completing the snapshot, the Debezium connectors occasionally encounter a transient exception (e.g. a network failure) when starting the streaming process. In some cases, if the offset for the last snapshot record is not flushed and the connector is restarted, it leads to re-snapshotting of the tables.

      To avoid this issue, this ticket proposes introducing a configurable delay before starting the streaming process after the snapshot is completed. 

      A new configuration parameter (`streaming.delay.ms`) can be added to control this delay. This is symmetrical to `snapshot.delay.ms` configuration that is already available to add delays before starting snapshotting.

      The default value would be set to `0`, which means there would be no delay by default. Users can configure a value slightly greater than the `offset.flush.interval.ms` configuration, ensuring that offsets are properly flushed before streaming begins.

      This adjustment should help prevent unnecessary re-snapshotting and improve the overall stability of the connector during the transition from snapshot to streaming.

              Unassigned Unassigned
              rdangwal Rajendra Dangwal (Inactive)
              Rajendra Dangwal (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: