Loading...

Type: Bug
Resolution: Done
Priority: Major
Fix Version/s: None
Affects Version/s: 1.3.0.Final
Component/s: postgresql-connector
Labels:
None

Blocked:
False
Ready:
False
Release Note Text:
Undefined
Steps to Reproduce:

Hide

Use both Single Message Transform configurations along with the Topic custom creation settings eg: retention settings.

_{{"name":"connectorTesting5",}
_"config":
_{
  _{"connector.class": "io.debezium.connector.postgresql.PostgresConnector",}
  _{"slot.name": "nfpbvcxy6gnzakz07gsr",}
  _{"tasks.max": "1",}
  _{"database.history.kafka.topic": "dbhistory.testdb",}
  _{"include.schema.changes": "true",}
  _{"table.whitelist": "public.table1, public.table2",}
  _{"decimal.handling.mode": "double",}
  _{"value.converter": "io.confluent.connect.avro.AvroConverter",}
  _{"key.converter": "io.confluent.connect.avro.AvroConverter",}
  _{"transforms":"unwrap",}
  _{"transforms.unwrap.type":"io.debezium.transforms.ExtractNewRecordState",}
  _{"transforms.unwrap.drop.tombstones": "false",}
  _{"transforms.unwrap.delete.handling.mode":"rewrite",}
  _{"database.user": "postgres",}
  _{"database.dbname": "testdb",}
  _{"database.history.kafka.bootstrap.servers": "http://kafka:9092",}
  _{"time.precision.mode": "connect",}
  _{"database.server.name": "kpi5.testdb",}
  _{"database.port": "8080",}
  _{"plugin.name": "pgoutput",}
  _{"value.converter.schema.registry.url": "http://schema-registry:8081",}
  _{"database.hostname": "localhost",}
  _{"database.password": "postgres",}
  _{"key.converter.schema.registry.url": "http://schema-registry:8081",}
  _{"topic.creation.groups": "tombs,config",}
  _{"topic.creation.tombs.include": "kpi5.testdb.public.table1, kpi5.testdb.public.table2",}
  _{"topic.creation.tombs.cleanup.policy": "compact",}
  _{"topic.creation.tombs.delete.retention.ms": 0,}
  _{"topic.creation.default.replication.factor":1,}
  _{"topic.creation.default.partitions": 1,}
  _{"topic.creation.default.cleanup.policy": "delete",}
  _{"topic.creation.default.compression.type": "lz4"}
_}
_}

Show
Use both Single Message Transform configurations along with the Topic custom creation settings eg: retention settings. {"name":"connectorTesting5", "config": {    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",    "slot.name": "nfpbvcxy6gnzakz07gsr",    "tasks.max": "1",    "database.history.kafka.topic": "dbhistory.testdb",    "include.schema.changes": "true",    "table.whitelist": "public.table1, public.table2",    "decimal.handling.mode": "double",    "value.converter": "io.confluent.connect.avro.AvroConverter",    "key.converter": "io.confluent.connect.avro.AvroConverter",    "transforms":"unwrap",    "transforms.unwrap.type":"io.debezium.transforms.ExtractNewRecordState",    "transforms.unwrap.drop.tombstones": "false",    "transforms.unwrap.delete.handling.mode":"rewrite",    "database.user": "postgres",    "database.dbname": "testdb",    "database.history.kafka.bootstrap.servers": "http://kafka:9092",    "time.precision.mode": "connect",    "database.server.name": "kpi5.testdb",    "database.port": "8080",    "plugin.name": "pgoutput",    "value.converter.schema.registry.url": "http://schema-registry:8081",    "database.hostname": "localhost",    "database.password": "postgres",    "key.converter.schema.registry.url": "http://schema-registry:8081",    "topic.creation.groups": "tombs,config",    "topic.creation.tombs.include": "kpi5.testdb.public.table1, kpi5.testdb.public.table2",       "topic.creation.tombs.cleanup.policy": "compact",    "topic.creation.tombs.delete.retention.ms": 0,    "topic.creation.default.replication.factor":1,    "topic.creation.default.partitions": 1,       "topic.creation.default.cleanup.policy": "delete",       "topic.creation.default.compression.type": "lz4" } }
Market:

SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

Debezium 1.3.0.Final , creating a connector for postgresdb to Kafka with SMT & topic custom creation settings combined - Connector gets created & then fails with the following error

_{2020-11-02 14:24:13,387 ERROR || Failed to start task connectorTesting5-0 [org.apache.kafka.connect.runtime.Worker]}
_{org.apache.kafka.connect.errors.ConnectException: org.apache.kafka.common.config.ConfigException: Unknown configuration 'transforms.unwrap.type'}
_{at org.apache.kafka.connect.runtime.ConnectorConfig.transformations(ConnectorConfig.java:296)}
_{at org.apache.kafka.connect.runtime.Worker.buildWorkerTask(Worker.java:605)}
_{at org.apache.kafka.connect.runtime.Worker.startTask(Worker.java:555)}
_{at org.apache.kafka.connect.runtime.distributed.DistributedHerder.startTask(DistributedHerder.java:1251)}
_{at org.apache.kafka.connect.runtime.distributed.DistributedHerder.access$1700(DistributedHerder.java:127)}
_{at org.apache.kafka.connect.runtime.distributed.DistributedHerder$10.call(DistributedHerder.java:1266)}
_{at org.apache.kafka.connect.runtime.distributed.DistributedHerder$10.call(DistributedHerder.java:1262)}
_{at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)}
_{at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)}
_{at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)}
_{at java.base/java.lang.Thread.run(Thread.java:834)}
_{Caused by: org.apache.kafka.common.config.ConfigException: Unknown configuration 'transforms.unwrap.type'}
_{at org.apache.kafka.common.config.AbstractConfig.get(AbstractConfig.java:159)}
_{at org.apache.kafka.connect.runtime.SourceConnectorConfig$EnrichedSourceConnectorConfig.get(SourceConnectorConfig.java:57)}
_{at org.apache.kafka.connect.runtime.SourceConnectorConfig.get(SourceConnectorConfig.java:141)}
_{at org.apache.kafka.common.config.AbstractConfig.getClass(AbstractConfig.java:216)}
_{at org.apache.kafka.connect.runtime.ConnectorConfig.transformations(ConnectorConfig.java:281)}
_{... 10 more}
As mentioned in one of the closed jiras, issue appears to be linked to https://issues.apache.org/jira/browse/KAFKA-10387 . Which was fixed in Kafka-Connect versions 2.6.1 and above. Debezium 1.0.3.Final currently appears to have Kafka-Connect
"version": "2.6.0"

duplicates

DBZ-2573 Unknown configuration 'transforms.<name>.type'

Closed

is related to

DBZ-2630 Upgrade to Apache Kafka Connect 2.6.1

Closed

Details

Description

Attachments

Issue Links

Activity

People

Dates