Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-2719

Connector fails on creation- SMT and Topic custom creation settings do not work together

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • None
    • 1.3.0.Final
    • postgresql-connector
    • None
    • False
    • False
    • Undefined
    • Hide

      Use both Single Message Transform configurations along with the Topic custom creation settings  eg: retention settings.

      {"name":"connectorTesting5",
      "config":
      {
        "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
        "slot.name": "nfpbvcxy6gnzakz07gsr",
        "tasks.max": "1",
        "database.history.kafka.topic": "dbhistory.testdb",
        "include.schema.changes": "true",
        "table.whitelist": "public.table1, public.table2",
        "decimal.handling.mode": "double",
        "value.converter": "io.confluent.connect.avro.AvroConverter",
        "key.converter": "io.confluent.connect.avro.AvroConverter",
        "transforms":"unwrap",
        "transforms.unwrap.type":"io.debezium.transforms.ExtractNewRecordState",
        "transforms.unwrap.drop.tombstones": "false",
        "transforms.unwrap.delete.handling.mode":"rewrite",
        "database.user": "postgres",
        "database.dbname": "testdb",
        "database.history.kafka.bootstrap.servers": "http://kafka:9092",
        "time.precision.mode": "connect",
        "database.server.name": "kpi5.testdb",
        "database.port": "8080",
        "plugin.name": "pgoutput",
        "value.converter.schema.registry.url": "http://schema-registry:8081",
        "database.hostname": "localhost",
        "database.password": "postgres",
        "key.converter.schema.registry.url": "http://schema-registry:8081",
        "topic.creation.groups": "tombs,config",
        "topic.creation.tombs.include": "kpi5.testdb.public.table1, kpi5.testdb.public.table2",  
        "topic.creation.tombs.cleanup.policy": "compact",
        "topic.creation.tombs.delete.retention.ms": 0,
        "topic.creation.default.replication.factor":1,
        "topic.creation.default.partitions": 1,  
        "topic.creation.default.cleanup.policy": "delete",  
        "topic.creation.default.compression.type": "lz4"
      }
      }

      Show
      Use both Single Message Transform configurations along with the Topic custom creation settings  eg: retention settings. {"name":"connectorTesting5", "config": {    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",    "slot.name": "nfpbvcxy6gnzakz07gsr",    "tasks.max": "1",    "database.history.kafka.topic": "dbhistory.testdb",    "include.schema.changes": "true",    "table.whitelist": "public.table1, public.table2",    "decimal.handling.mode": "double",    "value.converter": "io.confluent.connect.avro.AvroConverter",    "key.converter": "io.confluent.connect.avro.AvroConverter",    "transforms":"unwrap",    "transforms.unwrap.type":"io.debezium.transforms.ExtractNewRecordState",    "transforms.unwrap.drop.tombstones": "false",    "transforms.unwrap.delete.handling.mode":"rewrite",    "database.user": "postgres",    "database.dbname": "testdb",    "database.history.kafka.bootstrap.servers": "http://kafka:9092",    "time.precision.mode": "connect",    "database.server.name": "kpi5.testdb",    "database.port": "8080",    "plugin.name": "pgoutput",    "value.converter.schema.registry.url": "http://schema-registry:8081",    "database.hostname": "localhost",    "database.password": "postgres",    "key.converter.schema.registry.url": "http://schema-registry:8081",    "topic.creation.groups": "tombs,config",    "topic.creation.tombs.include": "kpi5.testdb.public.table1, kpi5.testdb.public.table2",       "topic.creation.tombs.cleanup.policy": "compact",    "topic.creation.tombs.delete.retention.ms": 0,    "topic.creation.default.replication.factor":1,    "topic.creation.default.partitions": 1,       "topic.creation.default.cleanup.policy": "delete",       "topic.creation.default.compression.type": "lz4" } }

      Debezium 1.3.0.Final , creating a connector for postgresdb to Kafka with SMT & topic custom creation settings combined - Connector gets created & then fails with the following error 

      2020-11-02 14:24:13,387 ERROR || Failed to start task connectorTesting5-0 [org.apache.kafka.connect.runtime.Worker]
      org.apache.kafka.connect.errors.ConnectException: org.apache.kafka.common.config.ConfigException: Unknown configuration 'transforms.unwrap.type'
      at org.apache.kafka.connect.runtime.ConnectorConfig.transformations(ConnectorConfig.java:296)
      at org.apache.kafka.connect.runtime.Worker.buildWorkerTask(Worker.java:605)
      at org.apache.kafka.connect.runtime.Worker.startTask(Worker.java:555)
      at org.apache.kafka.connect.runtime.distributed.DistributedHerder.startTask(DistributedHerder.java:1251)
      at org.apache.kafka.connect.runtime.distributed.DistributedHerder.access$1700(DistributedHerder.java:127)
      at org.apache.kafka.connect.runtime.distributed.DistributedHerder$10.call(DistributedHerder.java:1266)
      at org.apache.kafka.connect.runtime.distributed.DistributedHerder$10.call(DistributedHerder.java:1262)
      at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
      at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
      at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
      at java.base/java.lang.Thread.run(Thread.java:834)
      Caused by: org.apache.kafka.common.config.ConfigException: Unknown configuration 'transforms.unwrap.type'
      at org.apache.kafka.common.config.AbstractConfig.get(AbstractConfig.java:159)
      at org.apache.kafka.connect.runtime.SourceConnectorConfig$EnrichedSourceConnectorConfig.get(SourceConnectorConfig.java:57)
      at org.apache.kafka.connect.runtime.SourceConnectorConfig.get(SourceConnectorConfig.java:141)
      at org.apache.kafka.common.config.AbstractConfig.getClass(AbstractConfig.java:216)
      at org.apache.kafka.connect.runtime.ConnectorConfig.transformations(ConnectorConfig.java:281)
      ... 10 more
      As mentioned in one of the closed jiras, issue appears to be linked to https://issues.apache.org/jira/browse/KAFKA-10387 . Which was fixed in Kafka-Connect versions 2.6.1 and above. Debezium 1.0.3.Final currently appears to have Kafka-Connect 
      "version": "2.6.0" 

            Unassigned Unassigned
            nagendra.kotharu Nagendra Kotharu (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: