Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-101

Out of memory error when MySql result set is larger than 1000 (3 million rows)

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Blocker Blocker
    • None
    • 0.2.3
    • mysql-connector
    • None

    • Mac OSX (4CPU/8GB RAM) running debezium zookeeper, kafka and connect in a Docker host (virtual box 4GB RAM).

    • Hide

      The steps to reproduce are

      • Start ZooKeeper
      • Start Kafka
      • Start Connect
      • Add Connector
      • When a table exists with a large result-set > 1000 rows (3 million rows) the error shown above occurs.
      Show
      The steps to reproduce are Start ZooKeeper Start Kafka Start Connect Add Connector When a table exists with a large result-set > 1000 rows (3 million rows) the error shown above occurs.

      See below for the error including the stacktrace ...

      2016-08-15 05:51:00,729 ERROR MySQL|mysql-server-local|snapshot Failed due to error: Aborting snapshot after running 'SELECT * FROM optimis_development.contacts': Java heap space [io.debezium.connector.mysql.SnapshotReader]
      java.lang.OutOfMemoryError: Java heap space
      at com.mysql.jdbc.Buffer.<init>(Buffer.java:62)
      at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:2023)
      at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:3403)
      at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:470)
      at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:3105)
      at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:2336)
      at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2729)
      at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2545)
      at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2503)
      at com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1369)
      at io.debezium.jdbc.JdbcConnection.query(JdbcConnection.java:277)
      at io.debezium.connector.mysql.SnapshotReader.execute(SnapshotReader.java:311)
      at io.debezium.connector.mysql.SnapshotReader$$Lambda$132/1065165958.run(Unknown Source)
      at java.lang.Thread.run(Thread.java:745)
      2016-08-15 05:51:00,806 ERROR MySQL|mysql-server-local|task Task optimis-connector-0 threw an uncaught and unrecoverable exception [org.apache.kafka.connect.runtime.WorkerSourceTask]
      2016-08-15 05:51:00,806 ERROR MySQL|mysql-server-local|task Task is being killed and will not recover until manually restarted: [org.apache.kafka.connect.runtime.WorkerSourceTask]
      org.apache.kafka.connect.errors.ConnectException: Aborting snapshot after running 'SELECT * FROM optimis_development.contacts': Java heap space
      at io.debezium.connector.mysql.AbstractReader.failed(AbstractReader.java:118)
      at io.debezium.connector.mysql.SnapshotReader.execute(SnapshotReader.java:388)
      at java.lang.Thread.run(Thread.java:745)
      Caused by: java.lang.OutOfMemoryError: Java heap space
      at com.mysql.jdbc.Buffer.<init>(Buffer.java:62)
      at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:2023)
      at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:3403)
      at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:470)
      at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:3105)
      at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:2336)
      at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2729)
      at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2545)
      at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2503)
      at com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1369)
      at io.debezium.jdbc.JdbcConnection.query(JdbcConnection.java:277)
      at io.debezium.connector.mysql.SnapshotReader.execute(SnapshotReader.java:311)
      at io.debezium.connector.mysql.SnapshotReader$$Lambda$132/1065165958.run(Unknown Source)
      ... 1 more

              Unassigned Unassigned
              uchouhan Umang Chouhan (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: