Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-3331

Data loss when MongoDB snapshot take longer than the Oplog Window

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Major
    • 1.6.0.Beta1
    • 1.4.0.Final
    • mongodb-connector
    • None

    Description

      Overview:
      We are currently using the MongoDB Connector on a very large collection. The Oplog window is configured to be 12 hours but the snapshot took days. I expected the connector to fail when the snapshot completed but it started streaming happily with this cryptic log message.

      INFO The last event processed was not transactional, resuming at the oplog event after 'Timestamp{value=6940351246901450050, seconds=1615926448, inc=5442}
      

      I assume there is some data loss between the Snapshot and Streaming phase because the position in the Oplog streaming should have started is no longer in the Oplog?

      Also, can you explain the log message, would we get this ever if streaming started when the snapshot completed in the Oplog window? 

      Attachments

        Activity

          People

            ccranfor@redhat.com Chris Cranford
            ryan.tomczik Ryan Tomczik (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: