Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-94

Out of Memory when taking snapshot of large table

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Critical
    • 0.2.4, 0.3
    • 0.2.3
    • mysql-connector
    • None

    Description

      using debezium/connect:0.2 to source a large database
      Debezium throwed OOM when taking snapshot

      Running on Amazon EC2 with 16G memory

      2016-08-04 13:12:43,901 INFO   MySQL|mysql_host_52_70_200_92_port_3307|snapshot  Step 8.2: scanned table 'gpay.app_billing' in 00:01:25.56   [io.debezium.connector.mysql.SnapshotReader]
2016-08-04 13:14:00,793 ERROR  MySQL|mysql_host_52_70_200_92_port_3307|snapshot  Failed due to error: Aborting snapshot after running 'SELECT * FROM gpay.app_installation': Java heap space   [io.debezium.connector.mysql.SnapshotReader]
java.lang.OutOfMemoryError: Java heap space
	at com.mysql.jdbc.MysqlIO.nextRowFast(MysqlIO.java:2205)
	at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1984)
	at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:3403)
	at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:470)
	at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:3105)
	at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:2336)
	at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2729)
	at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2545)
	at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2503)
	at com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1369)
	at io.debezium.jdbc.JdbcConnection.query(JdbcConnection.java:277)
	at io.debezium.connector.mysql.SnapshotReader.execute(SnapshotReader.java:311)
	at io.debezium.connector.mysql.SnapshotReader$$Lambda$82/1145170159.run(Unknown Source)
	at java.lang.Thread.run(Thread.java:745)
2016-08-04 13:14:01,179 ERROR  MySQL|mysql_host_52_70_200_92_port_3307|task  Task gpay-0 threw an uncaught and unrecoverable exception   [org.apache.kafka.connect.runtime.WorkerSourceTask]
2016-08-04 13:14:01,180 ERROR  MySQL|mysql_host_52_70_200_92_port_3307|task  Task is being killed and will not recover until manually restarted:   [org.apache.kafka.connect.runtime.WorkerSourceTask]
org.apache.kafka.connect.errors.ConnectException: Aborting snapshot after running 'SELECT * FROM gpay.app_installation': Java heap space
	at io.debezium.connector.mysql.AbstractReader.failed(AbstractReader.java:118)
	at io.debezium.connector.mysql.SnapshotReader.execute(SnapshotReader.java:388)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Java heap space
	at com.mysql.jdbc.MysqlIO.nextRowFast(MysqlIO.java:2205)
	at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1984)
	at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:3403)
	at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:470)
	at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:3105)
	at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:2336)
	at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2729)
	at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2545)
	at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2503)
	at com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1369)
	at io.debezium.jdbc.JdbcConnection.query(JdbcConnection.java:277)
	at io.debezium.connector.mysql.SnapshotReader.execute(SnapshotReader.java:311)
	at io.debezium.connector.mysql.SnapshotReader$$Lambda$82/1145170159.run(Unknown Source)
	... 1 more

      table info:
      Name: app_installation
      Engine: InnoDB
      Version: 10
      Row_format: Dynamic
      Rows: 28267215
      Avg_row_length: 104
      Data_length: 2967470080
      Max_data_length: 0
      Index_length: 4715446272
      Data_free: 4194304
      Auto_increment: 29442980
      Create_time: 2016-08-04 05:07:17
      Update_time: NULL
      Check_time: NULL
      Collation: utf8_general_ci
      Checksum: NULL
      Create_options:
      Comment:

      Attachments

        Issue Links

          is duplicated by

          Bug - A problem that impairs or prevents the functions of the product. DBZ-101 Out of memory error when MySql result set is larger than 1000 (3 million rows)

          • Blocker - To be worked above all other priorities. This term is chosen when the severity of the issue is very high, has no workaround, or the effort for the change is comparatively low. Issues that may be very publicly visible and could generate significant media attention may also drive a higher priority.
          • Closed
          is related to

          Bug - A problem that impairs or prevents the functions of the product. DBZ-3309 Out of memory with mysql snapshots (regression of DBZ-94)

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed

          Activity

            People

              rhauch Randall Hauch (Inactive)
              wangshao_jira barten barten (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: