Details
-
Bug
-
Resolution: Done
-
Major
-
0.10.0.Final, 1.0.0.Beta2
-
None
-
- start Debezium to monitor a Postgresql 11+ database, using pgoutput plugin
- execute a truncate query on a monitored database
- at this point an exception is raised by Debezium
Description
Hi,
I'm working with Debezium embedded to capture chages from Postgresql databases.
I found an issue with pgoutput plugin and Postgresql 11+: when a truncate query is executed on a monitored table an exception is raised by Debezium. As a matter of fact, Postgresql 11+ sends streams also for truncate query; however they are not handled by PgOutputMessageDecoder.
This is the raised issue:
java.lang.IllegalArgumentException: Unsupported message type: T at io.debezium.connector.postgresql.connection.pgoutput.PgOutputMessageDecoder$MessageType.forType(PgOutputMessageDecoder.java:85) ~[debezium-connector-postgres-1.0.0.Beta2.jar:1.0.0.Beta2] at io.debezium.connector.postgresql.connection.pgoutput.PgOutputMessageDecoder.shouldMessageBeSkipped(PgOutputMessageDecoder.java:100) ~[debezium-connector-postgres-1.0.0.Beta2.jar:1.0.0.Beta2] at io.debezium.connector.postgresql.connection.PostgresReplicationConnection$1.readPending(PostgresReplicationConnection.java:408) ~[debezium-connector-postgres-1.0.0.Beta2.jar:1.0.0.Beta2] at io.debezium.connector.postgresql.PostgresStreamingChangeEventSource.execute(PostgresStreamingChangeEventSource.java:113) ~[debezium-connector-postgres-1.0.0.Beta2.jar:1.0.0.Beta2] at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:95) [debezium-core-1.0.0.Beta2.jar:1.0.0.Beta2] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_212] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_212] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_212]
This is a snapshot of the code of the class PgOutputMessageDecoder, which highlights that the truncate (type T) is not handled:
public enum MessageType { RELATION, BEGIN, COMMIT, INSERT, UPDATE, DELETE, TYPE, ORIGIN; public static MessageType forType(char type) { switch (type) { case 'R': return RELATION; case 'B': return BEGIN; case 'C': return COMMIT; case 'I': return INSERT; case 'U': return UPDATE; case 'D': return DELETE; case 'Y': return TYPE; case 'O': return ORIGIN; default: throw new IllegalArgumentException("Unsupported message type: " + type); } } }
These are my configurations:
connector.class=io.debezium.connector.postgresql.PostgresConnector offset.storage=org.apache.kafka.connect.storage.FileOffsetBackingStore offset.storage.file.filename=./offset.dat offset.flush.interval.ms=15000 name=test_cdc_pg_debezium database.hostname=localhost database.port=5432 plugin.name=pgoutput slot.name=test_cdc_pg_debezium_slot slot.drop_on_stop=false publication.name=test_cdc_pg_debezium_publication database.user=test database.password=******** database.dbname=my_database table.whitelist=public\\.my_partitioned_table_\\d{8} database.server.name=test_cdc_pg_debezium_server snapshot.mode=never tombstones.on.delete=false database.connectTimeout=30000 database.socketTimeout=30000
With Postgresql 10 there isn't this problem because truncate events are not emitted.
With wal2json plugin there are no problems also with Postgresql 11+.
I tested both the last stable version (0.10.0.Final) and the last development version (1.0.0.Beta2) of Debezium.
If you need more details please ask me.
Thank you
Alberto
Attachments
Issue Links
- relates to
-
DBZ-1052 Publish TX boundary markers on a TX metadata topic
-
- Closed
-
