Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-681

PostgreSQL connector doesn't handle TIME(p) columns correctly with wal2json

XMLWordPrintable

    • Hide

      1. Have a PostgeSQL instance with wal2Json plugin (I used docker debezium/postgres)
      2. Have a PostgreSQL connector stood up, my config:
      {
      "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
      "plugin.name": "wal2json",
      "tasks.max": "1",
      "database.hostname": "postgres",
      "database.port": "5432",
      "database.user": "postgres",
      "database.password": "***",
      "database.dbname": "postgres",
      "database.server.name": "elasticsearch",
      "table.whitelist": "public.dbo.testmssqltable",
      "decimal.handling.mode": "double"
      }
      3. create matching table:
      CREATE TABLE "dbo.testmssqltable" (
      "id" int not NULL,
      "int" int,
      "numeric" numeric,
      "time" time,
      PRIMARY KEY("id"));
      4. Insert example data, with time with precision:
      insert into "dbo.testmssqltable" ("id","int","numeric", "time") VALUES (1, 1, 111.234567, '12:12:12.1');
      5. Error is thrown and connectors stops to work, and I am actually unable to continue to work with this connector as long as there is such time example in DB, as it will fail each time when restarted.

      Show
      1. Have a PostgeSQL instance with wal2Json plugin (I used docker debezium/postgres) 2. Have a PostgreSQL connector stood up, my config: { "connector.class": "io.debezium.connector.postgresql.PostgresConnector", "plugin.name": "wal2json", "tasks.max": "1", "database.hostname": "postgres", "database.port": "5432", "database.user": "postgres", "database.password": "***", "database.dbname": "postgres", "database.server.name": "elasticsearch", "table.whitelist": "public.dbo.testmssqltable", "decimal.handling.mode": "double" } 3. create matching table: CREATE TABLE "dbo.testmssqltable" ( "id" int not NULL, "int" int, "numeric" numeric, "time" time, PRIMARY KEY("id")); 4. Insert example data, with time with precision: insert into "dbo.testmssqltable" ("id","int","numeric", "time") VALUES (1, 1, 111.234567, '12:12:12.1'); 5. Error is thrown and connectors stops to work, and I am actually unable to continue to work with this connector as long as there is such time example in DB, as it will fail each time when restarted.

      When using PostgresSQL source connector on a table with a column of type Time, connector will fail (and stop working) when there is example data present with precision.

      Stack trace:

      org.apache.kafka.connect.errors.ConnectException: An exception ocurred in the change event producer. This connector will be stopped.
	at io.debezium.connector.base.ChangeEventQueue.throwProducerFailureIfPresent(ChangeEventQueue.java:168)
	at io.debezium.connector.base.ChangeEventQueue.poll(ChangeEventQueue.java:149)
	at io.debezium.connector.postgresql.PostgresConnectorTask.poll(PostgresConnectorTask.java:151)
	at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:179)
	at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:170)
	at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:214)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.kafka.connect.errors.ConnectException: java.time.format.DateTimeParseException: Text '12:12:12.1' could not be parsed, unparsed text found at index 8
	at io.debezium.connector.postgresql.connection.wal2json.DateTimeFormat$ISODateTimeFormat.format(DateTimeFormat.java:117)
	at io.debezium.connector.postgresql.connection.wal2json.DateTimeFormat$ISODateTimeFormat.time(DateTimeFormat.java:92)
	at io.debezium.connector.postgresql.connection.wal2json.Wal2JsonReplicationMessage.getValue(Wal2JsonReplicationMessage.java:250)
	at io.debezium.connector.postgresql.connection.wal2json.Wal2JsonReplicationMessage$1.getValue(Wal2JsonReplicationMessage.java:135)
	at io.debezium.connector.postgresql.RecordsStreamProducer.lambda$columnValues$2(RecordsStreamProducer.java:430)
	at java.util.ArrayList.forEach(ArrayList.java:1249)
	at io.debezium.connector.postgresql.RecordsStreamProducer.columnValues(RecordsStreamProducer.java:425)
	at io.debezium.connector.postgresql.RecordsStreamProducer.process(RecordsStreamProducer.java:239)
	at io.debezium.connector.postgresql.RecordsStreamProducer.lambda$streamChanges$1(RecordsStreamProducer.java:119)
	at io.debezium.connector.postgresql.connection.wal2json.Wal2JsonMessageDecoder.processMessage(Wal2JsonMessageDecoder.java:61)
	at io.debezium.connector.postgresql.connection.PostgresReplicationConnection$1.deserializeMessages(PostgresReplicationConnection.java:209)
	at io.debezium.connector.postgresql.connection.PostgresReplicationConnection$1.read(PostgresReplicationConnection.java:194)
	at io.debezium.connector.postgresql.RecordsStreamProducer.streamChanges(RecordsStreamProducer.java:119)
	at io.debezium.connector.postgresql.RecordsStreamProducer.lambda$start$0(RecordsStreamProducer.java:105)
	... 5 more
Caused by: java.time.format.DateTimeParseException: Text '12:12:12.1' could not be parsed, unparsed text found at index 8
	at java.time.format.DateTimeFormatter.parseResolved0(DateTimeFormatter.java:1952)
	at java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:1851)
	at java.time.LocalTime.parse(LocalTime.java:441)
	at io.debezium.connector.postgresql.connection.wal2json.DateTimeFormat$ISODateTimeFormat.lambda$time$2(DateTimeFormat.java:92)
	at io.debez

              jpechane Jiri Pechanec
              przemekak Przemek Pawlowski (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: