-
Bug
-
Resolution: Done
-
Critical
-
0.2.3
-
None
using debezium/connect:0.2 to source a large database
Debezium throwed OOM when taking snapshot
Running on Amazon EC2 with 16G memory
2016-08-04 13:12:43,901 INFO MySQL|mysql_host_52_70_200_92_port_3307|snapshot Step 8.2: scanned table 'gpay.app_billing' in 00:01:25.56 [io.debezium.connector.mysql.SnapshotReader] 2016-08-04 13:14:00,793 ERROR MySQL|mysql_host_52_70_200_92_port_3307|snapshot Failed due to error: Aborting snapshot after running 'SELECT * FROM gpay.app_installation': Java heap space [io.debezium.connector.mysql.SnapshotReader] java.lang.OutOfMemoryError: Java heap space at com.mysql.jdbc.MysqlIO.nextRowFast(MysqlIO.java:2205) at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1984) at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:3403) at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:470) at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:3105) at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:2336) at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2729) at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2545) at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2503) at com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1369) at io.debezium.jdbc.JdbcConnection.query(JdbcConnection.java:277) at io.debezium.connector.mysql.SnapshotReader.execute(SnapshotReader.java:311) at io.debezium.connector.mysql.SnapshotReader$$Lambda$82/1145170159.run(Unknown Source) at java.lang.Thread.run(Thread.java:745) 2016-08-04 13:14:01,179 ERROR MySQL|mysql_host_52_70_200_92_port_3307|task Task gpay-0 threw an uncaught and unrecoverable exception [org.apache.kafka.connect.runtime.WorkerSourceTask] 2016-08-04 13:14:01,180 ERROR MySQL|mysql_host_52_70_200_92_port_3307|task Task is being killed and will not recover until manually restarted: [org.apache.kafka.connect.runtime.WorkerSourceTask] org.apache.kafka.connect.errors.ConnectException: Aborting snapshot after running 'SELECT * FROM gpay.app_installation': Java heap space at io.debezium.connector.mysql.AbstractReader.failed(AbstractReader.java:118) at io.debezium.connector.mysql.SnapshotReader.execute(SnapshotReader.java:388) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.OutOfMemoryError: Java heap space at com.mysql.jdbc.MysqlIO.nextRowFast(MysqlIO.java:2205) at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1984) at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:3403) at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:470) at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:3105) at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:2336) at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2729) at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2545) at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2503) at com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1369) at io.debezium.jdbc.JdbcConnection.query(JdbcConnection.java:277) at io.debezium.connector.mysql.SnapshotReader.execute(SnapshotReader.java:311) at io.debezium.connector.mysql.SnapshotReader$$Lambda$82/1145170159.run(Unknown Source) ... 1 more
table info:
Name: app_installation
Engine: InnoDB
Version: 10
Row_format: Dynamic
Rows: 28267215
Avg_row_length: 104
Data_length: 2967470080
Max_data_length: 0
Index_length: 4715446272
Data_free: 4194304
Auto_increment: 29442980
Create_time: 2016-08-04 05:07:17
Update_time: NULL
Check_time: NULL
Collation: utf8_general_ci
Checksum: NULL
Create_options:
Comment: