Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-6027

MongoDB Connector snapshot.fetch.size doesn't work

    XMLWordPrintable

Details

    • Bug
    • Resolution: Not a Bug
    • Major
    • None
    • 2.1.1.Final
    • mongodb-connector
    • None
    • False
    • None
    • False
    • Closing this as it looks like everything works correctly. Please reopen the issues if there are any additional information available.
    • Important

    Description

      In order to make your issue reports as actionable as possible, please provide the following information, depending on the issue type.

      Bug report

      For bug reports, provide this information, please:

      What Debezium connector do you use and what version?

      2.1.1 Final

      What is the connector configuration?

       
       

      max.queue.size: 81290   
      max.batch.size: 10000   
      mongodb.ssl.enabled: true
      poll.interval.ms: 40
      snapshot.max.threads: 60 
      snapshot.fetch.size: 40000  

      What are the captured database version and mode of deployment?

      (E.g. on-premises, with a specific cloud provider, etc.)

      MongoDB Atlas 4.2 

       

      What behavior do you expect?

      I expect that the connector read more from the MongoDB when I increase the fetch size

      What behavior do you see?

      It doesn't matter how much I increase the snapshot.fetch.size, it wouldn't increase the read rate from MongoDB. The snapshot queue is always almost empty (based on the remaining capacity metrics) and the producers are operating as fast as the input rate.

      It seems that this configuration is either not correctly set or is ignored by the primary node.

      Do you see the same behavior using the latest released Debezium version?

      (Ideally, also verify with the latest Alpha/Beta/CR version)

      This is the latest version

      Do you have the connector logs, ideally from the start till the finish?

      (You might be asked later to provide DEBUG/TRACE level log)

      I couldn't find any related information in the log but please let me know if I can add anything. 

      How to reproduce the issue using our tutorial deployment?

      Run the debezium connector with the above configuration or perhaps set the snapshot.fetch.size to a very large number. You wouldn't see any changes in the number of read operations on MongoDB.

      Attachments

        Activity

          People

            jcechace@redhat.com Jakub Čecháček
            thr.firoozian@gmail.com Tahereh Firoozian (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: