Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-681

PostgreSQL connector doesn't handle TIME(p) columns correctly with wal2json

XMLWordPrintable

    • Hide

      1. Have a PostgeSQL instance with wal2Json plugin (I used docker debezium/postgres)
      2. Have a PostgreSQL connector stood up, my config:
      {
      "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
      "plugin.name": "wal2json",
      "tasks.max": "1",
      "database.hostname": "postgres",
      "database.port": "5432",
      "database.user": "postgres",
      "database.password": "***",
      "database.dbname": "postgres",
      "database.server.name": "elasticsearch",
      "table.whitelist": "public.dbo.testmssqltable",
      "decimal.handling.mode": "double"
      }
      3. create matching table:
      CREATE TABLE "dbo.testmssqltable" (
      "id" int not NULL,
      "int" int,
      "numeric" numeric,
      "time" time,
      PRIMARY KEY("id"));
      4. Insert example data, with time with precision:
      insert into "dbo.testmssqltable" ("id","int","numeric", "time") VALUES (1, 1, 111.234567, '12:12:12.1');
      5. Error is thrown and connectors stops to work, and I am actually unable to continue to work with this connector as long as there is such time example in DB, as it will fail each time when restarted.

      Show
      1. Have a PostgeSQL instance with wal2Json plugin (I used docker debezium/postgres) 2. Have a PostgreSQL connector stood up, my config: { "connector.class": "io.debezium.connector.postgresql.PostgresConnector", "plugin.name": "wal2json", "tasks.max": "1", "database.hostname": "postgres", "database.port": "5432", "database.user": "postgres", "database.password": "***", "database.dbname": "postgres", "database.server.name": "elasticsearch", "table.whitelist": "public.dbo.testmssqltable", "decimal.handling.mode": "double" } 3. create matching table: CREATE TABLE "dbo.testmssqltable" ( "id" int not NULL, "int" int, "numeric" numeric, "time" time, PRIMARY KEY("id")); 4. Insert example data, with time with precision: insert into "dbo.testmssqltable" ("id","int","numeric", "time") VALUES (1, 1, 111.234567, '12:12:12.1'); 5. Error is thrown and connectors stops to work, and I am actually unable to continue to work with this connector as long as there is such time example in DB, as it will fail each time when restarted.

      When using PostgresSQL source connector on a table with a column of type Time, connector will fail (and stop working) when there is example data present with precision.

      Stack trace:

      org.apache.kafka.connect.errors.ConnectException: An exception ocurred in the change event producer. This connector will be stopped.
      	at io.debezium.connector.base.ChangeEventQueue.throwProducerFailureIfPresent(ChangeEventQueue.java:168)
      	at io.debezium.connector.base.ChangeEventQueue.poll(ChangeEventQueue.java:149)
      	at io.debezium.connector.postgresql.PostgresConnectorTask.poll(PostgresConnectorTask.java:151)
      	at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:179)
      	at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:170)
      	at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:214)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: org.apache.kafka.connect.errors.ConnectException: java.time.format.DateTimeParseException: Text '12:12:12.1' could not be parsed, unparsed text found at index 8
      	at io.debezium.connector.postgresql.connection.wal2json.DateTimeFormat$ISODateTimeFormat.format(DateTimeFormat.java:117)
      	at io.debezium.connector.postgresql.connection.wal2json.DateTimeFormat$ISODateTimeFormat.time(DateTimeFormat.java:92)
      	at io.debezium.connector.postgresql.connection.wal2json.Wal2JsonReplicationMessage.getValue(Wal2JsonReplicationMessage.java:250)
      	at io.debezium.connector.postgresql.connection.wal2json.Wal2JsonReplicationMessage$1.getValue(Wal2JsonReplicationMessage.java:135)
      	at io.debezium.connector.postgresql.RecordsStreamProducer.lambda$columnValues$2(RecordsStreamProducer.java:430)
      	at java.util.ArrayList.forEach(ArrayList.java:1249)
      	at io.debezium.connector.postgresql.RecordsStreamProducer.columnValues(RecordsStreamProducer.java:425)
      	at io.debezium.connector.postgresql.RecordsStreamProducer.process(RecordsStreamProducer.java:239)
      	at io.debezium.connector.postgresql.RecordsStreamProducer.lambda$streamChanges$1(RecordsStreamProducer.java:119)
      	at io.debezium.connector.postgresql.connection.wal2json.Wal2JsonMessageDecoder.processMessage(Wal2JsonMessageDecoder.java:61)
      	at io.debezium.connector.postgresql.connection.PostgresReplicationConnection$1.deserializeMessages(PostgresReplicationConnection.java:209)
      	at io.debezium.connector.postgresql.connection.PostgresReplicationConnection$1.read(PostgresReplicationConnection.java:194)
      	at io.debezium.connector.postgresql.RecordsStreamProducer.streamChanges(RecordsStreamProducer.java:119)
      	at io.debezium.connector.postgresql.RecordsStreamProducer.lambda$start$0(RecordsStreamProducer.java:105)
      	... 5 more
      Caused by: java.time.format.DateTimeParseException: Text '12:12:12.1' could not be parsed, unparsed text found at index 8
      	at java.time.format.DateTimeFormatter.parseResolved0(DateTimeFormatter.java:1952)
      	at java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:1851)
      	at java.time.LocalTime.parse(LocalTime.java:441)
      	at io.debezium.connector.postgresql.connection.wal2json.DateTimeFormat$ISODateTimeFormat.lambda$time$2(DateTimeFormat.java:92)
      	at io.debez
      

            jpechane Jiri Pechanec
            przemekak Przemek Pawlowski (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: