-
Bug
-
Resolution: Done
-
Critical
-
0.6.2
-
None
When Debezium encounters an insert/update/delete binlog event that it cannot process, it appears to skip the event entirely with only a Debug level log message. This causes not only data loss, but lack of visibility that it encountered an issue.
Data Loss: Since the event is skipped, the reader continues to read the binlog beyond this point and commit future offsets. Therefore, it may be impossible to recover the skipped events which may have been valid indeed.
Visibility The skipped events are only logged at Debug level, so they are not visible in production logs. When we encountered this on production environment, it took 4 weeks to notice something was wrong.
For use cases that require no data loss, we need the reader to stop completely and throw a hard exception if it encounters this scenario. This gives clear notice and a chance to figure out what went wrong and attempt to recover.
Example: For UPDATE event, this is the only log message we currently get. Note this is DEBUG level only, nothing on INFO level. [2018-01-18 16:52:03,920] DEBUG Skipping update table metadata event: Event{header=EventHeaderV4{timestamp=1516312323000, eventType=TABLE_MAP, serverId=223344, headerLength=19, dataLength=100, nextPosition=6456, flags=0}, data=TableMapEventData{tableId=444, database='mydb', table='mytable', columnTypes=-2, 15, 15, 15, 15, 15, -2, -2, 10, 1, 1, -2, -2, -2, 15, 15, -2, 17, -2, 17, -2, 17, columnMetadata=65056, 50, 50, 50, 255, 100, 63233, 65056, 0, 0, 0, 63233, 63233, 65026, 32, 32, 65056, 0, 65056, 0, 65056, 0, columnNullability={5, 6, 8, 11, 12}}} (io.debezium.connector.mysql.BinlogReader:633) ]}} (io.debezium.connector.mysql.BinlogReader:730)