-
Bug
-
Resolution: Done
-
Major
-
2.5.2.Final
-
None
-
False
-
None
-
False
What Debezium connector do you use and what version?
PostgreSQL, v2.5.2.Final**
This DBZ-7582 Correctly serialize toasted HSTORE values when unavailable we've merged into our v2.5.2.Final build
What is the connector configuration?
{
"connector.class": "io.debezium.connector.postgresql.PostgresConnector",
"errors.log.include.messages": "true",
"publication.name": "streamkap_pub",
"transforms": "unwrap",
"schema.include.list": "public",
"signal.enabled.channels": "source,kafka",
"predicates.IgnoreHeartbeatStateExtraction.type": "org.apache.kafka.connect.transforms.predicates.TopicNameMatches",
"predicates.IgnoreHeartbeatStateExtraction.pattern": ".*-heartbeat.*",
"database.sslmode": "require",
"unavailable.value.placeholder": "{\"__streamkap_unavailable_value\": true}",
"transforms.unwrap.drop.tombstones": "true",
"signal.data.collection": "public.streamkap_signal",
"transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState",
"errors.log.enable": "true",
"snapshot.fetch.size": "102400",
"key.converter": "io.confluent.connect.avro.AvroConverter",
"database.user": "...",
"database.dbname": "...",
"signal.kafka.bootstrap.servers": "kafka-0:9094,kafka-1:9094,kafka-2:9094",
"signal.kafka.topic": "source_65d3.streamkap_signal",
"heartbeat.interval.ms": "300000",
"transforms.unwrap.field.name.adjustment.mode": "avro",
"plugin.name": "pgoutput",
"value.converter.schema.registry.url": "http://schema-registry:8081",
"predicates": "IgnoreHeartbeatStateExtraction",
"transforms.unwrap.add.fields.prefix": "_streamkap_",
"producer.override.max.request.size": "67108864",
"errors.max.retries": "5",
"database.password": "...",
"name": "source_65d3",
"errors.tolerance": "all",
"max.batch.size": "4096",
"snapshot.mode": "never",
"transforms.unwrap.delete.handling.mode": "rewrite",
"max.queue.size": "204800",
"slot.name": "streamkap_pgoutput_slot",
"incremental.snapshot.chunk.size": "102400",
"hstore.handling.mode": "json",
"producer.override.batch.size": "102400",
"transforms.unwrap.predicate": "IgnoreHeartbeatStateExtraction",
"topic.prefix": "source_65d3",
"binary.handling.mode": "bytes",
"value.converter": "io.confluent.connect.avro.AvroConverter",
"snapshot.max.threads": "1",
"transforms.unwrap.add.headers": "op,db",
"database.port": "5432",
"transforms.unwrap.negate": "true",
"database.hostname": "aurora-cluster...rds.amazonaws.com",
"schema.name.adjustment.mode": "avro",
"transforms.unwrap.add.fields": "source.ts_ms,ts_ms",
"table.include.list": "...",
"key.converter.schema.registry.url": "http://schema-registry:8081",
"post.processors": "reselector",
"reselector.type": "io.debezium.processors.reselect.ReselectColumnsPostProcessor",
"reselector.reselect.columns.include.list": "...",
"reselector.reselect.unavailable.values": "true",
"reselector.reselect.null.values": "true",
"reselector.reselect.use.event.key": "false"
}
What is the captured database version and mode of depoyment?
PostgreSQL 15.4, AWS PostgreSQL Aurora cluster. Table replica identities are all DEFAULT
What behaviour do you expect?
Toasted values to be replaced with the unavailable.value.placeholder and then re-selected by the post processor.
What behaviour do you see?
The Re-select Post Processor does not work for toasted values of columns with complex types like HSTORE. It consistently reports in debug logs:
No columns require re-selection.
The toasted value won't be re-selected because the placeholder value format for these HSTORE columns is a string representation of JSON like this:
{ "placeholder value" : "placeholder value" }
Where placeholder value is replaced by unavailable.value.placeholder. This fails the equality check in the isUnavailableValueHolder method.
How to reproduce the issue using our tutorial deployment?
No, but I have reproduced locally.
- links to
-
RHEA-2024:129636
Red Hat build of Debezium 2.5.4 release
