-
Bug
-
Resolution: Unresolved
-
Major
-
0.9.5.Final
Am using Debezium to stream data from MariaDB to Kafka.
During the initial load the data is copied properly but the database changes are detected by Debezium but not sent to kafka.
Below is error stack trace. Any leads highly appreciated.
Sep 13 08:08:19 ip-172-31-4-198 connect-distributed[4062]: [2019-09-13 08:08:19,859] ERROR Encountered change event 'Event{header=EventHeaderV4{timestamp=1568361089000, eventType=TABLE_MAP, serverId=1, headerLength=19, dataLength=89, nextPosition=4919121, flags=0}, data=TableMapEventData{tableId=2697, database='reports', table='TABLE_NAME', columnTypes=3, 3, -2, 3, 3, 3, 18, 5, 5, 5, 18, 18, 3, 5, 5, 5, 5, 5, 5, 5, 5, -2, 3, 5, 3, 3, 3, 3, 3, 18, columnMetadata=0, 0, 65072, 0, 0, 0, 0, 8, 8, 8, 0, 0, 0, 8, 8, 8, 8, 8, 8, 8, 8, 65120, 0, 8, 0, 0, 0, 0, 0, 0, columnNullability={}, eventMetadata=null}}' at offset {ts_sec=1568361089, file=mariadb-bin.000209, pos=4918332, server_id=1, event=2} for table reports.TABLE_NAME whose schema isn't known to this connector. One possible cause is an incomplete database history topic. Take a new snapshot in this case. Sep 13 08:08:19 ip-172-31-4-198 connect-distributed[4062]: Use the mysqlbinlog tool to view the problematic event: mysqlbinlog --start-position=4919013 --stop-position=4919121 --verbose mariadb-bin.000209 (io.debezium.connector.mysql.BinlogReader:747) Sep 13 08:08:19 ip-172-31-4-198 connect-distributed[4062]: [2019-09-13 08:08:19,859] ERROR Error during binlog processing. Last offset stored = null, binlog reader near position = mariadb-bin.000209/4919013 (io.debezium.connector.mysql.BinlogReader:1054) Sep 13 08:08:19 ip-172-31-4-198 connect-distributed[4062]: [2019-09-13 08:08:19,859] ERROR Failed due to error: Error processing binlog event (io.debezium.connector.mysql.BinlogReader:209) Sep 13 08:08:19 ip-172-31-4-198 connect-distributed[4062]: org.apache.kafka.connect.errors.ConnectException: Encountered change event for table reports.TABLE_NAMEwhose schema isn't known to this connector Sep 13 08:08:19 ip-172-31-4-198 connect-distributed[4062]: #011at io.debezium.connector.mysql.AbstractReader.wrap(AbstractReader.java:230) Sep 13 08:08:19 ip-172-31-4-198 connect-distributed[4062]: #011at io.debezium.connector.mysql.AbstractReader.failed(AbstractReader.java:208) Sep 13 08:08:19 ip-172-31-4-198 connect-distributed[4062]: #011at io.debezium.connector.mysql.BinlogReader.handleEvent(BinlogReader.java:508) Sep 13 08:08:19 ip-172-31-4-198 connect-distributed[4062]: #011at com.github.shyiko.mysql.binlog.BinaryLogClient.notifyEventListeners(BinaryLogClient.java:1130) Sep 13 08:08:19 ip-172-31-4-198 connect-distributed[4062]: #011at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:978) Sep 13 08:08:19 ip-172-31-4-198 connect-distributed[4062]: #011at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:581) Sep 13 08:08:19 ip-172-31-4-198 connect-distributed[4062]: #011at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:860) Sep 13 08:08:19 ip-172-31-4-198 connect-distributed[4062]: #011at java.lang.Thread.run(Thread.java:748) Sep 13 08:08:19 ip-172-31-4-198 connect-distributed[4062]: Caused by: org.apache.kafka.connect.errors.ConnectException: Encountered change event for table reports.TABLE_NAMEwhose schema isn't known to this connector Sep 13 08:08:19 ip-172-31-4-198 connect-distributed[4062]: #011at io.debezium.connector.mysql.BinlogReader.informAboutUnknownTableIfRequired(BinlogReader.java:758) Sep 13 08:08:19 ip-172-31-4-198 connect-distributed[4062]: #011at io.debezium.connector.mysql.BinlogReader.handleUpdateTableMetadata(BinlogReader.java:733) Sep 13 08:08:19 ip-172-31-4-198 connect-distributed[4062]: #011at io.debezium.connector.mysql.BinlogReader.handleEvent(BinlogReader.java:492) Sep 13 08:08:19 ip-172-31-4-198 connect-distributed[4062]: #011... 5 more Sep 13 08:08:19 ip-172-31-4-198 connect-distributed[4062]: [2019-09-13 08:08:19,859] INFO Error processing binlog event, and propagating to Kafka Connect so it stops this connector. Future binlog events read before connector is shutdown will be ignored. (io.debezium.connector.mysql.BinlogReader:513) Sep 13 08:08:20 ip-172-31-4-198 connect-distributed[4062]: [2019-09-13 08:08:20,134] INFO WorkerSourceTask{id=mysql_live_2-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:398) Sep 13 08:08:20 ip-172-31-4-198 connect-distributed[4062]: [2019-09-13 08:08:20,134] INFO WorkerSourceTask{id=mysql_live_2-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:415) Sep 13 08:08:20 ip-172-31-4-198 connect-distributed[4062]: [2019-09-13 08:08:20,135] INFO WorkerSourceTask{id=mysql_live_2-0} Finished commitOffsets successfully in 1 ms (org.apache.kafka.connect.runtime.WorkerSourceTask:497) Sep 13 08:08:20 ip-172-31-4-198 connect-distributed[4062]: [2019-09-13 08:08:20,135] ERROR WorkerSourceTask{id=mysql_live_2-0} Task threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask:179) Sep 13 08:08:20 ip-172-31-4-198 connect-distributed[4062]: org.apache.kafka.connect.errors.ConnectException: Encountered change event for table reports.TABLE_NAMEwhose schema isn't known to this connector Sep 13 08:08:20 ip-172-31-4-198 connect-distributed[4062]: #011at io.debezium.connector.mysql.AbstractReader.wrap(AbstractReader.java:230) Sep 13 08:08:20 ip-172-31-4-198 connect-distributed[4062]: #011at io.debezium.connector.mysql.AbstractReader.failed(AbstractReader.java:208) Sep 13 08:08:20 ip-172-31-4-198 connect-distributed[4062]: #011at io.debezium.connector.mysql.BinlogReader.handleEvent(BinlogReader.java:508) Sep 13 08:08:20 ip-172-31-4-198 connect-distributed[4062]: #011at com.github.shyiko.mysql.binlog.BinaryLogClient.notifyEventListeners(BinaryLogClient.java:1130) Sep 13 08:08:20 ip-172-31-4-198 connect-distributed[4062]: #011at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:978) Sep 13 08:08:20 ip-172-31-4-198 connect-distributed[4062]: #011at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:581) Sep 13 08:08:20 ip-172-31-4-198 connect-distributed[4062]: #011at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:860) Sep 13 08:08:20 ip-172-31-4-198 connect-distributed[4062]: #011at java.lang.Thread.run(Thread.java:748) Sep 13 08:08:20 ip-172-31-4-198 connect-distributed[4062]: Caused by: org.apache.kafka.connect.errors.ConnectException: Encountered change event for table reports.TABLE_NAMEwhose schema isn't known to this connector Sep 13 08:08:20 ip-172-31-4-198 connect-distributed[4062]: #011at io.debezium.connector.mysql.BinlogReader.informAboutUnknownTableIfRequired(BinlogReader.java:758) Sep 13 08:08:20 ip-172-31-4-198 connect-distributed[4062]: #011at io.debezium.connector.mysql.BinlogReader.handleUpdateTableMetadata(BinlogReader.java:733) Sep 13 08:08:20 ip-172-31-4-198 connect-distributed[4062]: #011at io.debezium.connector.mysql.BinlogReader.handleEvent(BinlogReader.java:492) Sep 13 08:08:20 ip-172-31-4-198 connect-distributed[4062]: #011... 5 more Sep 13 08:08:20 ip-172-31-4-198 connect-distributed[4062]: [2019-09-13 08:08:20,136] ERROR WorkerSourceTask{id=mysql_live_2-0} Task is being killed