Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-9386

Setting a transaction retention period can incorrectly resolve the transaction cache's eldest SCN

XMLWordPrintable

    • False
    • Hide

      None

      Show
      None
    • False

      In the buffered implementation method calculateNewStartScn, if the transaction cache has any entries, one of the first actions this method does is to purge any transactions that have aged beyond the transaction retention period by calling abandonTransactions.

      The first problem is that abandonTransactions uses the last processed SCN to move backward to compute a threshold SCN where any transactions that begin before this position should be discarded.

      If the cache has values

      • OldestScnDetails is properly set
      • abandonTransactions explicitly sets offset SCN as threshold rather than Oldest SCN

      The calculateNewStartScn then changes the offset SCN in the following ways:

      • LOB enabled
        • If cache is empty, sets SCN to the last committed SCN if available; otherwise unchanged.
        • If cache had values, sets the SCN to the eldest SCN in the cache, but this is the eldest SCN from prior to the transaction abandonment. This should recalculate the minimum SCN if transactions were abandoned.
      • LOB not enabled
        • SET will be left to the eldest SCN prior to the abandonment and is only advanced if the cache is empty.

      Overall this logic is far too brittle and should be changed as follows:

      • If transactions are abandoned, minCacheScn should be recomputed.
      • Cache removals should be based on recalculated minCacheScn.

      This should not have any impact on the connector's function, other than it will put some added unnecessary processing stress when LOB is enabled since the calculateNewStartScn returns the currently set offset SCN and causes extra reprocessing when the cache isn't empty. For non LOB configurations, this would only create issues if the connector was stopped and restarted immediately after a batch. It would force the first mining iteration to re-read and discard transactions on the first mining iteration that had previously been abandoned.

              ccranfor@redhat.com Chris Cranford
              ccranfor@redhat.com Chris Cranford
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated: