Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-5966

Truncate records incompatible with ExtractNewRecordState

    XMLWordPrintable

Details

    • False
    • None
    • False

    Description

      Bug report

      What Debezium connector do you use and what version?

      debezium postgres connector
      v1.9.3

      What is the connector configuration?

      ```

      {     "connector.class": "io.debezium.connector.postgresql.PostgresConnector",     "tasks.max": "1",     "database.hostname": "postgres",     "database.port": "5432",     "database.user": "myuser",     "database.password": "mypassword",     "database.dbname": "postgres",     "database.server.name": "asgard",     "plugin.name": "pgoutput",     "truncate.handling.mode": "include",     "transforms": "unwrap",     "transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState",     "transforms.unwrap.drop.tombstones": "false",     "transforms.unwrap.delete.handling.mode": "rewrite",     "transforms.unwrap.add.fields": "op, lsn",     "_transforms.unwrap.predicate": "IsTombstone",     "_transforms.unwrap.negate": "false",     "predicates": "IsTombstone",     "predicates.IsTombstone.type": "org.apache.kafka.connect.transforms.predicates.RecordIsTombstone" }

      ```

      What is the captured database version and mode of deployment?

      (E.g. on-premises, with a specific cloud provider, etc.)

      ```

                                                                 version
      -----------------------------------------------------------------------------------------------------------------------------
       PostgreSQL 15.1 (Debian 15.1-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit

      ```

      What behaviour do you expect?

      Working connector with truncate events coming along with fields mentioned in `transforms.unwrap.add.fields`.

      What behaviour do you see?

      Connector fails with the following stack trace as soon as we issue the truncate command on the concerned db table.

      ```

      [2023-01-01 14:40:18,454] ERROR [debezium-postgres-source|task-0] WorkerSourceTask{id=debezium-postgres-source-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask:208)
      org.apache.kafka.connect.errors.ConnectException: Tolerance exceeded in error handler
          at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:237)
          at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execute(RetryWithToleranceOperator.java:159)
          at org.apache.kafka.connect.runtime.TransformationChain.transformRecord(TransformationChain.java:70)
          at org.apache.kafka.connect.runtime.TransformationChain.apply(TransformationChain.java:50)
          at org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.sendRecords(AbstractWorkerSourceTask.java:389)
          at org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.execute(AbstractWorkerSourceTask.java:357)
          at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:201)
          at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:256)
          at org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.run(AbstractWorkerSourceTask.java:75)
          at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
          at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
          at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
          at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
          at java.base/java.lang.Thread.run(Thread.java:829)
      Caused by: org.apache.kafka.connect.errors.DataException: Only Struct objects supported for [source field insertion], found: null
          at org.apache.kafka.connect.transforms.util.Requirements.requireStruct(Requirements.java:52)
          at io.debezium.transforms.ExtractNewRecordState.addFields(ExtractNewRecordState.java:228)
          at io.debezium.transforms.ExtractNewRecordState.apply(ExtractNewRecordState.java:165)
          at org.apache.kafka.connect.runtime.TransformationChain.lambda$transformRecord$0(TransformationChain.java:70)
          at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndRetry(RetryWithToleranceOperator.java:183)
          at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:217)
          ... 13 more

      ```

      Do you see the same behaviour using the latest relesead Debezium version?

      (Ideally, also verify with latest Alpha/Beta/CR version)

      Yes

      Do you have the connector logs, ideally from start till finish?

      (You might be asked later to provide DEBUG/TRACE level log)

      Attaching here connect-logs.txt

      How to reproduce the issue using our tutorial deployment?

      <Your answer>

      Feature request or enhancement

      For feature requests or enhancements, provide this information, please:

      Which use case/requirement will be addressed by the proposed feature?

      <Your answer>

      Implementation ideas (optional)

      <Your answer>

      Attachments

        Activity

          People

            Unassigned Unassigned
            akanimesh7 Animesh Kumar
            Votes:
            1 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: