Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-8977

Move unbuffered resume position calculation to secondary thread

XMLWordPrintable

    • Icon: Enhancement Enhancement
    • Resolution: Unresolved
    • Icon: Major Major
    • 3.4.0.Alpha1
    • None
    • oracle-connector
    • None
    • False
    • Hide

      None

      Show
      None
    • False

      In DBZ-8924 we introduced the new unbuffered LogMiner implementation. This implementation relies on secondary LogMiner session that recomputes the low watermark read position periodically. This computation can be intensive and could take several minutes to complete depending on the volume of activity since the current resume position.

      I'd like to propose we move this calculation to a secondary thread, introduce some state machine aspects to avoid race conditions, allowing the main thread to request a recalculation on the given interval, and only starting it if its not in-progress, and only providing the updated information to the main thread when requested.

      So in practice there would be a few states:

      • Ready
        This state signals that if the timer has expired, the resume position will trigger the background thread to compute the value. The handler will transition into an Inprogress state until the background thread computes the value.
      • Inprogress
        This state means the background thread is running, and any call to the handler should continue to return the current resume position, as if the timer has not yet expired.
      • Waiting
        This state means that the background thread has computed the resume position and is waiting on the main thread to call the handler. When the main thread does, it will return the computed value, resets the timer, and transitions back into the Ready state.

      This has various benefits, but primarily avoids blocking the main change event stream while the recalculation step runs. But this will introduce slightly more load on the database given that we have 2 LogMiner sessions that are running concurrently per connector.

              Unassigned Unassigned
              ccranfor@redhat.com Chris Cranford
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated: