Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-6412

MongoDB incremental snapshot start from beginning if an error is raised

    XMLWordPrintable

Details

    • False
    • None
    • False
    • Important

    Description

      Bug report

      For bug reports, provide this information, please:

      What Debezium connector do you use and what version?

      I use 5eb6f87b97aa19ba59cec1cddbb6f1ec8d2ba0ff commit from the debezium repo

      What is the connector configuration?

          signal.data.collection: <signal-collection-name>
          incremental.snapshot.chunk.size: <chunk-size>

      What is the captured database version and mode of deployment?

      MongoDB atlas 4.4

      Deployed on strimzi Kafka connector

      What behavior do you expect?

      The incremental snapshot fails because a document size is larger than the kafka permitted size. I updated the configuration on the kafka side to allow larger events and I expect the snapshot to resume from it failed. 

      I think the same behavior exists if the error is different. 

      What behavior do you see?

      The incremental snapshot starts from the beginning 

      Do you see the same behavior using the latest released Debezium version?

      Haven't tested it 

      Do you have the connector logs, ideally from start to finish?

      (You might be asked later to provide DEBUG/TRACE level log)

      Please let me know if I can provide any specific log to help debug this issue 

      How to reproduce the issue using our tutorial deployment?

      Generate a large message in the MongoDB ( you need to wait until this message is not in the oplog anymore or you can alternatively reduce the max message size in the connector or kafka to a very small number), run the connector, trigger the incremental snapshot and wait for the connector to fail.  

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              thr.firoozian@gmail.com Tahereh Firoozian (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated: