Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-8168

Postgres JSONB Fields are not supported with Reselect Post Processor

XMLWordPrintable

    • False
    • Hide

      None

      Show
      None
    • False

      Bug report

      What Debezium connector do you use and what version?

      2.7.1.Final

      What is the connector configuration?

       

      {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.hostname": "postgres",
    "database.port": "5432",
    "database.user": "postgres",
    "database.password": "postgres",
    "database.dbname": "test_database",
    "database.server.name": "test",
    "plugin.name": "pgoutput",
    "table.include.list": "test_table",
    "signal.data.collection": "public.debezium_signals",
    "slot.name": "debezium_test",
    "post.processors": "reselector",
    "reselector.type": "io.debezium.processors.reselect.ReselectColumnsPostProcessor",
    "reselect.columns.include.list": "public.test_table:jsonb_field,public.test_table:text_field",
    "reselect.unavailable.values": "true",
    "reselect.null.values": "false",
    "reselect.use.event.key": "false",
    "topic.prefix": "prefix"
}

       

       

      What is the captured database version and mode of deployment?

      Postgres 11-3.1 on docker

      What behavior do you expect?

      JSONB data to be reselected, instead of having "__debezium_unavailable_value" as a value

      What behavior do you see?

      TOASTed JSONB never reselected, with or without a default value set, also with or without NOT NULL constraint .. always shows "__debezium_unavailable_value"

      Do you see the same behaviour using the latest released Debezium version?

      Issue exists on both 2.7.1.FINAL and 3.0.0.Alpha2

      Do you have the connector logs, ideally from start till finish?

       

      2024-08-21 15:32:31,741 DEBUG  Postgres|prefix|streaming  Relation '124334' resolved to table 'public.test_table'   [io.debezium.connector.postgresql.PostgresSchema]
2024-08-21 15:32:31,742 DEBUG  Postgres|prefix|streaming  Adding column text_field for table public.test_table to re-select list due to unavailable value placeholder.   [io.debezium.processors.reselect.ReselectColumnsPostProcessor]
2024-08-21 15:32:32,142 DEBUG  ||  Sending 1 records to topic prefix.public.test_table   [io.debezium.connector.common.BaseSourceTask]
2024-08-21 15:32:32,142 INFO   ||  1 records sent during previous 00:03:17.9, last recorded offset of {server=prefix} partition is {lsn_proc=618714264, messageType=UPDATE, lsn_commit=618713600, lsn=618714264, txId=63695, ts_usec=1724254351326042}   [io.debezium.connector.common.BaseSourceTask]

      How to reproduce the issue using our tutorial deployment?

      • Create a postrges table with the following schema:
      CREATE TABLE public.test_table (
    id serial4 NOT NULL,
    jsonb_field jsonb DEFAULT '{}'::jsonb NOT NULL,
    text_field text NULL,
    dummy_field text null,
    CONSTRAINT test_table_pkey PRIMARY KEY (id)
);
      • Insert a record with more than 4kb data in the json_field/text_field, SAMPLE:
      https://pastebin.com/fMckNXvp
      • Add new connector with the previously mentioned configuration
      • Update the record and monitor the value of the fields
      update test_table set dummy_field = '123' where id = 1
      • Full value from kafka
      https://pastebin.com/5mknHL68

              Unassigned Unassigned
              moe.shaer Mohamed Shaer (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: