Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-4799

Incremental snapshotting in multi-partition mode

    XMLWordPrintable

Details

    • Enhancement
    • Resolution: Unresolved
    • Major
    • 2.4-backlog
    • None
    • sqlserver-connector
    • None
    • False
    • False

    Description

      The incremental snapshotting feature introduced in Incremental snapshotting DBZ-3500 will not work as the end-users might expect with the multi-partition connector configuration being introduced in DBZ-2975.

      There are the following challenges:

      1. In the original design, the connector processes a single database, and the signaling table resides in the same database. It allows the connector to receive signals from the same database that it captures and store its state by updating the signaling table (the LSNs of the updates in the signaling table will exist in the same "address space" as the LSNs of the captured changes). With the multi-partition configuration, even if there is a single task, a single signaling table cannot properly capture the state of the snapshotting process because its LSNs will belong to at most one (or even none) of the databases being captured.
      2. If the connector runs multiple tasks, then a single signaling table will be captured by one of the tasks. The task that receives the signals will not be able to handle the signals related to the databases that it doesn't capture.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              sergeimorozov Sergei Morozov
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated: