Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-9683

MongoDB connector: Use wallTime for source.ts_ms accuracy

XMLWordPrintable

    • Icon: Enhancement Enhancement
    • Resolution: Done
    • Icon: Minor Minor
    • 3.4.0.Beta1
    • None
    • mongodb-connector
    • None

      Bug report

      MongoDB connector should use wallTime instead of clusterTime for source.ts_ms in change stream events to accurately represent the database operation time.

      What Debezium connector do you use and what version?

      According to Debezium documentation, source.ts_ms should indicate "the time that the change was made in the database." For MongoDB 6.0+ change streams, source.ts_ms should use wallTime (actual wall-clock time of the database operation) rather than clusterTime (logical oplog timestamp).

      Also this ensures accurate lag calculation (ts_ms - source.ts_ms) by measuring the time between the actual database change and Debezium processing.

      What is the connector configuration?

      NA

      What is the captured database version and mode of deployment?

      Mongo 8.0.15, onprem

      What behavior do you expect?

      source.ts_ms should use wallTime from MongoDB change streams

      What behavior do you see?

      source.ts_ms shows clusterTime from MongoDB change streams

      Do you see the same behaviour using the latest released Debezium version?

      Yes

      Do you have the connector logs, ideally from start till finish?

      NA

      Implementation ideas (optional)

      1. Modify SourceInfo.timestamp() to prefer wallTime over clusterTime when available
      1. Fall back to clusterTime when wallTime is not available (MongoDB < 6.0 or when not provided)
      1. The wallTime field is already being extracted from ChangeStreamDocument in SourceInfo.changeStreamEvent()

              Unassigned Unassigned
              sumitjha4321@gmail.com Sumit Jha
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated:
                Resolved: