Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-7421

Poor performance with incremental snapshot with long list of tables

XMLWordPrintable

    • False
    • None
    • False

      Bug report

      What Debezium connector do you use and what version?

      Mysql Connector - Debezium 2.4.0

      What is the issue?

      When an incremental snapshot is running, and a lot of input binlog messages are being processed in parallel, the processing performance can become very poor.

      After profiling the application, this is correlated to the size of the "dataCollectionsToSnapshot" data structure. In a nutshell, the more tables you snapshot, the worse the performance will get.

      Profiling results:

      As seen in the results, this is due to the call to "dataCollectionsToSnapshotAsString" for each new binlog records in case a snapshot is running.

       

       

      This might explain issue DBZ-7256

        1. image-2024-01-27-17-20-08-585.png
          image-2024-01-27-17-20-08-585.png
          268 kB

            [DBZ-7421] Poor performance with incremental snapshot with long list of tables

            Errata Tool added a comment -

            Since the problem described in this issue should be resolved in a recent advisory, it has been closed.

            For information on the advisory (Red Hat build of Debezium 2.5.4 release), and where to find the updated files, follow the link below.

            If the solution does not work for you, open a new bug report.
            https://access.redhat.com/errata/RHEA-2024:1726

            Errata Tool added a comment - Since the problem described in this issue should be resolved in a recent advisory, it has been closed. For information on the advisory (Red Hat build of Debezium 2.5.4 release), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2024:1726

            Drew von Zweck added a comment -

            maximethebault Hello, I have some questions about your implementation and success with Incremental Snapshotting, sent you an email. Thanks.

            Drew von Zweck added a comment - maximethebault Hello, I have some questions about your implementation and success with Incremental Snapshotting, sent you an email. Thanks.

            Debezium Builder added a comment -

            Released

            Debezium Builder added a comment - Released

            Mario Fiore Vitale added a comment -

            Happy to hear that. Thanks to you for reporting it. 

            Mario Fiore Vitale added a comment - Happy to hear that. Thanks to you for reporting it. 

            Maxime Thebault added a comment -

            Thanks, I can confirm the issue is fixed in that snapshot version. Thanks for fixing it so quickly!

            Maxime Thebault added a comment - Thanks, I can confirm the issue is fixed in that snapshot version. Thanks for fixing it so quickly!

            Mario Fiore Vitale added a comment -

            You can get the 2.6 snapshot from here

            Mario Fiore Vitale added a comment - You can get the 2.6 snapshot from here

            Maxime Thebault added a comment -

            Hi rh-ee-mvitale 
            I can give it a try. Are there instructions anywhere on how I can find the snapshot artifacts?

            Maxime Thebault added a comment - Hi rh-ee-mvitale   I can give it a try. Are there instructions anywhere on how I can find the snapshot artifacts?

            Mario Fiore Vitale added a comment -

            Hi maximethebault , a feedback on the fix will be very appreciated. 

            Mario Fiore Vitale added a comment - Hi maximethebault , a feedback on the fix will be very appreciated. 

            Jiri Pechanec added a comment -

            Applied to 2.5 and 2.6

            Jiri Pechanec added a comment - Applied to 2.5 and 2.6

            Maxime Thebault added a comment -

            Hi!
            On my HW, If snapshotting around 260 tables, binlog processing saturates at about 1k messages/second.
            If the number of tables doubles, the binlog processing saturates at about 500 messages/second. This seems linear.

            Maxime Thebault added a comment - Hi! On my HW, If snapshotting around 260 tables, binlog processing saturates at about 1k messages/second. If the number of tables doubles, the binlog processing saturates at about 500 messages/second. This seems linear.

              rh-ee-mvitale Mario Fiore Vitale
              maximethebault Maxime Thebault
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated:
                Resolved: