Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-996

ArrayIndexOutOfBoundsException when a column is deleted (Postgres)

    Details

      Description

      We are using Debezium 0.8.3.Final and we are running into an ArrayIndexOutOfBoundsException.

      The following setup is triggering the exception:

      Message that is parsed:

      {
        "xid" : 7083777,
        "timestamp" : "2018-11-17 05:19:41.915029+00",
        "change" : [ {
          "kind" : "update",
          "schema" : "SCHEMA",
          "table" : "TABLE",
          "columnnames" : [ "COLUMN_A", "COLUMN_B", "COLUMN_C", "COLUMN_D" ],
          "columntypes" : [ "numeric(19,0)", "character varying(40)", "character varying(40)", "character varying(40)" ],
          "columnvalues" : [ 33, "UUID_1", "UUID_2", "UUID_3" ],
          "oldkeys" : {
            "keynames" : [ "COLUMN_A", "COLUMN_B", "COLUMN_C" ],
            "keytypes" : [ "numeric(19,0)", "character varying(40)", "character varying(40)" ],
            "keyvalues" : [ 33, "UUID_1", "UUID_2" ]
          }
        }]}
      

      Actual Database Schema
      SCHEMA.TABLE with three columns: COLUM_A, COLUMN_B, COLUMN_D

      The primary key has been changed from (COLUMN_A, COLUMN_C) to (COLUMN_A, COLUMN_D).

      Thrown exception

      java.lang.ArrayIndexOutOfBoundsException: -1
        at io.debezium.connector.postgresql.RecordsStreamProducer.columnValues(RecordsStreamProducer.java:441)
        at io.debezium.connector.postgresql.RecordsStreamProducer.process(RecordsStreamProducer.java:251)
        at io.debezium.connector.postgresql.RecordsStreamProducer.lambda$streamChanges$2(RecordsStreamProducer.java:128)
        at io.debezium.connector.postgresql.connection.wal2json.NonStreamingWal2JsonMessageDecoder.processMessage(NonStreamingWal2JsonMessageDecoder.java:62)
        at io.debezium.connector.postgresql.connection.PostgresReplicationConnection$1.deserializeMessages(PostgresReplicationConnection.java:218)
        at io.debezium.connector.postgresql.connection.PostgresReplicationConnection$1.read(PostgresReplicationConnection.java:203)
        at io.debezium.connector.postgresql.RecordsStreamProducer.streamChanges(RecordsStreamProducer.java:128)
        at io.debezium.connector.postgresql.RecordsStreamProducer.lambda$start$1(RecordsStreamProducer.java:114)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
      

      Cause: (RecordsStreamProducer)

              // based on the schema columns, create the values on the same position as the columns
              List<String> columnNames = table.columnNames();
              // JSON does not deliver a list of all columns for REPLICA IDENTITY DEFAULT
              Object[] values = new Object[columns.size() < columnNames.size() ? columnNames.size() : columns.size()];
      
              for (ReplicationMessage.Column column : columns) {
                  //DBZ-298 Quoted column names will be sent like that in messages, but stored unquoted in the column names
                  String columnName = Strings.unquoteIdentifierPart(column.getName());
                  int position = columnNames.indexOf(columnName);
                  assert position >= 0;
                  values[position] = column.getValue(this::typeResolverConnection, taskContext.config().includeUnknownDatatypes());
              }
      
      • COLUMN_C is processed in the for loop
      • position becomes -1
      • The assertion is not triggered since the -ea option is not provided by the Connect Docker image and assertions are by default disabled at runtime in Java

      However, it is a valid case that a column is removed from a table so throwing an AssertionError would have been a bug as well.

      Notes:

      • I came across https://issues.jboss.org/browse/DBZ-842 in my research where the ArrayIndexOutOfBoundsException is also mentioned in the comments. While my issue has a different cause. The fact that the ArrayIndexOutOfBoundsException can be thrown there is also because assertions are disabled at runtime.

        Gliffy Diagrams

          Attachments

            Activity

              People

              • Assignee:
                jpechanec Jiri Pechanec
                Reporter:
                wouter.bancken.aca Wouter Bancken
              • Votes:
                1 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: