Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-7596

Re-select Post Processor not working for complex types

XMLWordPrintable

      What Debezium connector do you use and what version?

      PostgreSQL, v2.5.2.Final**

      This DBZ-7582 Correctly serialize toasted HSTORE values when unavailable we've merged into our v2.5.2.Final build

      What is the connector configuration?

      {
        "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
        "errors.log.include.messages": "true",
        "publication.name": "streamkap_pub",
        "transforms": "unwrap",
        "schema.include.list": "public",
        "signal.enabled.channels": "source,kafka",
        "predicates.IgnoreHeartbeatStateExtraction.type": "org.apache.kafka.connect.transforms.predicates.TopicNameMatches",
        "predicates.IgnoreHeartbeatStateExtraction.pattern": ".*-heartbeat.*",
        "database.sslmode": "require",
        "unavailable.value.placeholder": "{\"__streamkap_unavailable_value\": true}",
        "transforms.unwrap.drop.tombstones": "true",
        "signal.data.collection": "public.streamkap_signal",
        "transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState",
        "errors.log.enable": "true",
        "snapshot.fetch.size": "102400",
        "key.converter": "io.confluent.connect.avro.AvroConverter",
        "database.user": "...",
        "database.dbname": "...",
        "signal.kafka.bootstrap.servers": "kafka-0:9094,kafka-1:9094,kafka-2:9094",
        "signal.kafka.topic": "source_65d3.streamkap_signal",
        "heartbeat.interval.ms": "300000",
        "transforms.unwrap.field.name.adjustment.mode": "avro",
        "plugin.name": "pgoutput",
        "value.converter.schema.registry.url": "http://schema-registry:8081",
        "predicates": "IgnoreHeartbeatStateExtraction",
        "transforms.unwrap.add.fields.prefix": "_streamkap_",
        "producer.override.max.request.size": "67108864",
        "errors.max.retries": "5",
        "database.password": "...",
        "name": "source_65d3",
        "errors.tolerance": "all",
        "max.batch.size": "4096",
        "snapshot.mode": "never",
        "transforms.unwrap.delete.handling.mode": "rewrite",
        "max.queue.size": "204800",
        "slot.name": "streamkap_pgoutput_slot",
        "incremental.snapshot.chunk.size": "102400",
        "hstore.handling.mode": "json",
        "producer.override.batch.size": "102400",
        "transforms.unwrap.predicate": "IgnoreHeartbeatStateExtraction",
        "topic.prefix": "source_65d3",
        "binary.handling.mode": "bytes",
        "value.converter": "io.confluent.connect.avro.AvroConverter",
        "snapshot.max.threads": "1",
        "transforms.unwrap.add.headers": "op,db",
        "database.port": "5432",
        "transforms.unwrap.negate": "true",
        "database.hostname": "aurora-cluster...rds.amazonaws.com",
        "schema.name.adjustment.mode": "avro",
        "transforms.unwrap.add.fields": "source.ts_ms,ts_ms",
        "table.include.list": "...",
        "key.converter.schema.registry.url": "http://schema-registry:8081",
        "post.processors": "reselector",
        "reselector.type": "io.debezium.processors.reselect.ReselectColumnsPostProcessor",
        "reselector.reselect.columns.include.list": "...",
        "reselector.reselect.unavailable.values": "true",
        "reselector.reselect.null.values": "true",
        "reselector.reselect.use.event.key": "false"
      }

      What is the captured database version and mode of depoyment?

      PostgreSQL 15.4, AWS PostgreSQL Aurora cluster. Table replica identities are all DEFAULT

      What behaviour do you expect?

      Toasted values to be replaced with the unavailable.value.placeholder and then re-selected by the post processor.

      What behaviour do you see?

      The Re-select Post Processor does not work for toasted values of columns with complex types like HSTORE. It consistently reports in debug logs:

      No columns require re-selection.

      The toasted value won't be re-selected because the placeholder value format for these HSTORE columns is a string representation of JSON like this:

      { "placeholder value" : "placeholder value" }
      

      Where placeholder value is replaced by unavailable.value.placeholder. This fails the equality check in the isUnavailableValueHolder method.

      How to reproduce the issue using our tutorial deployment?

      No, but I have reproduced locally.

            ccranfor@redhat.com Chris Cranford
            jonathono Jonathon Ogden
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: