Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-1576

Postgresql 11+ pgoutput plugin error with truncate

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: 0.10.0.Final, 1.0.0.Beta2
    • Fix Version/s: 1.0.0.Beta3
    • Component/s: postgresql-connector
    • Labels:
      None
    • Environment:

      Postgresql 11+
      pgoutput plugin

    • Steps to Reproduce:
      • start Debezium to monitor a Postgresql 11+ database, using pgoutput plugin
      • execute a truncate query on a monitored database
      • at this point an exception is raised by Debezium

      Description

      Hi,
      I'm working with Debezium embedded to capture chages from Postgresql databases.

      I found an issue with pgoutput plugin and Postgresql 11+: when a truncate query is executed on a monitored table an exception is raised by Debezium. As a matter of fact, Postgresql 11+ sends streams also for truncate query; however they are not handled by PgOutputMessageDecoder.

      This is the raised issue:

      java.lang.IllegalArgumentException: Unsupported message type: T
      	at io.debezium.connector.postgresql.connection.pgoutput.PgOutputMessageDecoder$MessageType.forType(PgOutputMessageDecoder.java:85) ~[debezium-connector-postgres-1.0.0.Beta2.jar:1.0.0.Beta2]
      	at io.debezium.connector.postgresql.connection.pgoutput.PgOutputMessageDecoder.shouldMessageBeSkipped(PgOutputMessageDecoder.java:100) ~[debezium-connector-postgres-1.0.0.Beta2.jar:1.0.0.Beta2]
      	at io.debezium.connector.postgresql.connection.PostgresReplicationConnection$1.readPending(PostgresReplicationConnection.java:408) ~[debezium-connector-postgres-1.0.0.Beta2.jar:1.0.0.Beta2]
      	at io.debezium.connector.postgresql.PostgresStreamingChangeEventSource.execute(PostgresStreamingChangeEventSource.java:113) ~[debezium-connector-postgres-1.0.0.Beta2.jar:1.0.0.Beta2]
      	at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:95) [debezium-core-1.0.0.Beta2.jar:1.0.0.Beta2]
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_212]
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_212]
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_212]
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_212]
      	at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_212]
      

      This is a snapshot of the code of the class PgOutputMessageDecoder, which highlights that the truncate (type T) is not handled:

          public enum MessageType {
              RELATION,
              BEGIN,
              COMMIT,
              INSERT,
              UPDATE,
              DELETE,
              TYPE,
              ORIGIN;
      
              public static MessageType forType(char type) {
                  switch (type) {
                      case 'R': return RELATION;
                      case 'B': return BEGIN;
                      case 'C': return COMMIT;
                      case 'I': return INSERT;
                      case 'U': return UPDATE;
                      case 'D': return DELETE;
                      case 'Y': return TYPE;
                      case 'O': return ORIGIN;
                      default: throw new IllegalArgumentException("Unsupported message type: " + type);
                  }
              }
          }
      

      These are my configurations:

      connector.class=io.debezium.connector.postgresql.PostgresConnector
      offset.storage=org.apache.kafka.connect.storage.FileOffsetBackingStore
      offset.storage.file.filename=./offset.dat
      offset.flush.interval.ms=15000
      name=test_cdc_pg_debezium
      database.hostname=localhost
      database.port=5432
      plugin.name=pgoutput
      slot.name=test_cdc_pg_debezium_slot
      slot.drop_on_stop=false
      publication.name=test_cdc_pg_debezium_publication
      database.user=test
      database.password=********
      database.dbname=my_database
      table.whitelist=public\\.my_partitioned_table_\\d{8}
      database.server.name=test_cdc_pg_debezium_server
      snapshot.mode=never
      tombstones.on.delete=false
      database.connectTimeout=30000
      database.socketTimeout=30000
      

      With Postgresql 10 there isn't this problem because truncate events are not emitted.

      With wal2json plugin there are no problems also with Postgresql 11+.

      I tested both the last stable version (0.10.0.Final) and the last development version (1.0.0.Beta2) of Debezium.

      If you need more details please ask me.

      Thank you
      Alberto

        Gliffy Diagrams

          Attachments

            Issue Links

              Activity

                People

                • Assignee:
                  ccranfor Chris Cranford
                  Reporter:
                  alucinip Alberto Lucini
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  4 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: