-
Bug
-
Resolution: Done
-
Major
-
0.8.3.Final
-
None
We are using Debezium 0.8.3.Final and we are running into an ArrayIndexOutOfBoundsException.
The following setup is triggering the exception:
Message that is parsed:
{
"xid" : 7083777,
"timestamp" : "2018-11-17 05:19:41.915029+00",
"change" : [ {
"kind" : "update",
"schema" : "SCHEMA",
"table" : "TABLE",
"columnnames" : [ "COLUMN_A", "COLUMN_B", "COLUMN_C", "COLUMN_D" ],
"columntypes" : [ "numeric(19,0)", "character varying(40)", "character varying(40)", "character varying(40)" ],
"columnvalues" : [ 33, "UUID_1", "UUID_2", "UUID_3" ],
"oldkeys" : {
"keynames" : [ "COLUMN_A", "COLUMN_B", "COLUMN_C" ],
"keytypes" : [ "numeric(19,0)", "character varying(40)", "character varying(40)" ],
"keyvalues" : [ 33, "UUID_1", "UUID_2" ]
}
}]}
Actual Database Schema
SCHEMA.TABLE with three columns: COLUM_A, COLUMN_B, COLUMN_D
The primary key has been changed from (COLUMN_A, COLUMN_C) to (COLUMN_A, COLUMN_D).
Thrown exception
java.lang.ArrayIndexOutOfBoundsException: -1 at io.debezium.connector.postgresql.RecordsStreamProducer.columnValues(RecordsStreamProducer.java:441) at io.debezium.connector.postgresql.RecordsStreamProducer.process(RecordsStreamProducer.java:251) at io.debezium.connector.postgresql.RecordsStreamProducer.lambda$streamChanges$2(RecordsStreamProducer.java:128) at io.debezium.connector.postgresql.connection.wal2json.NonStreamingWal2JsonMessageDecoder.processMessage(NonStreamingWal2JsonMessageDecoder.java:62) at io.debezium.connector.postgresql.connection.PostgresReplicationConnection$1.deserializeMessages(PostgresReplicationConnection.java:218) at io.debezium.connector.postgresql.connection.PostgresReplicationConnection$1.read(PostgresReplicationConnection.java:203) at io.debezium.connector.postgresql.RecordsStreamProducer.streamChanges(RecordsStreamProducer.java:128) at io.debezium.connector.postgresql.RecordsStreamProducer.lambda$start$1(RecordsStreamProducer.java:114) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
Cause: (RecordsStreamProducer)
// based on the schema columns, create the values on the same position as the columns List<String> columnNames = table.columnNames(); // JSON does not deliver a list of all columns for REPLICA IDENTITY DEFAULT Object[] values = new Object[columns.size() < columnNames.size() ? columnNames.size() : columns.size()]; for (ReplicationMessage.Column column : columns) { //DBZ-298 Quoted column names will be sent like that in messages, but stored unquoted in the column names String columnName = Strings.unquoteIdentifierPart(column.getName()); int position = columnNames.indexOf(columnName); assert position >= 0; values[position] = column.getValue(this::typeResolverConnection, taskContext.config().includeUnknownDatatypes()); }
- COLUMN_C is processed in the for loop
- position becomes -1
- The assertion is not triggered since the -ea option is not provided by the Connect Docker image and assertions are by default disabled at runtime in Java
However, it is a valid case that a column is removed from a table so throwing an AssertionError would have been a bug as well.
Notes:
- I came across https://issues.jboss.org/browse/DBZ-842 in my research where the ArrayIndexOutOfBoundsException is also mentioned in the comments. While my issue has a different cause. The fact that the ArrayIndexOutOfBoundsException can be thrown there is also because assertions are disabled at runtime.
