Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-3331

Data loss when MongoDB snapshot take longer than the Oplog Window

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • 1.6.0.Beta1
    • 1.4.0.Final
    • mongodb-connector
    • None

      Overview:
      We are currently using the MongoDB Connector on a very large collection. The Oplog window is configured to be 12 hours but the snapshot took days. I expected the connector to fail when the snapshot completed but it started streaming happily with this cryptic log message.

      INFO The last event processed was not transactional, resuming at the oplog event after 'Timestamp{value=6940351246901450050, seconds=1615926448, inc=5442}
      

      I assume there is some data loss between the Snapshot and Streaming phase because the position in the Oplog streaming should have started is no longer in the Oplog?

      Also, can you explain the log message, would we get this ever if streaming started when the snapshot completed in the Oplog window? 

              ccranfor@redhat.com Chris Cranford
              ryan.tomczik Ryan Tomczik (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated:
                Resolved: