Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-5556

Debezium Postgres Connector throwing invalid Java Object Error

    XMLWordPrintable

Details

    • Bug
    • Resolution: Unresolved
    • Major
    • 2.4-backlog
    • 1.9.5.Final
    • postgresql-connector
    • None
    • False
    • None
    • False
    • Hide
      • Postgres version is 10.12
      • The column is of type text array
      •  Create a table with multiple columns, including a column of type _text
      • Insert a row with a very large value for the _text column (it should get toasted)
      • Execute an update for that particular row, but for any other column other than the _text column
      • Debezium will throw the error, and will place a null value in the debezium message for the _text column, instead of the toasted value placeholder
      Show
      Postgres version is 10.12 The column is of type text array  Create a table with multiple columns, including a column of type _text Insert a row with a very large value for the _text column (it should get toasted) Execute an update for that particular row, but for any other column other than the _text column Debezium will throw the error, and will place a null value in the debezium message for the _text column, instead of the toasted value placeholder

    Description

      In order to make your issue reports as actionable as possible, please provide the following information, depending on the issue type.

      Bug report

      What Debezium connector do you use and what version?

      Postgres Connector v1.9.5.Final

      What is the connector configuration?

      connector.class: "io.debezium.connector.postgresql.PostgresConnector"
      database.history.file.filename: "/home/appRunner/debezium-publishers/dbhistory-test.dat"
      database.dbname: "redacted"
      database.user: "redacted"
      database.port: "5432"
      database.hostname: "redacted"
      database.password: "redacted"
      converter.schemas.enable: "false"
      offset.storage: "org.apache.kafka.connect.storage.FileOffsetBackingStore"
      database.server.name: "debezium-test-connector"
      plugin.name: "pgoutput"
      offset.flush.interval.ms: "1000"
      offset.storage.file.filename: "/home/appRunner/debezium-publishers/offsets-test.dat"
      name: "debezium-publisher-test"
      database.history: "io.debezium.relational.history.FileDatabaseHistory"
      snapshot.mode: "never"
      publication.name: "redacted"
      slot.name: "redacted"
      publication.autocreate.mode: "disabled"
      decimal.handling.mode: "double"
      binary.handling.mode: "hex"
      time.precision.mode: "adaptive_time_microseconds"

      What is the captured database version and mode of depoyment?

      Postgres 10.12 deployed in a cloudsql instance (GCP)

      What behaviour do you expect?

      The publisher should be able to parse all columns in the row and send the message with the appropriate value

      What behaviour do you see?

      Occasionally the publisher throws the following error 

      org.apache.kafka.connect.errors.DataException: Invalid Java object for schema with type ARRAY: class java.lang.String for field: "ipAddresses"

       and that particular field is assigned null value in the debezium message (i think)

      the column is of type _text

      The column value displayed in the logs is just a generic java object

       

      java.lang.Object@3777dc16 

      This happens when there's a very large value in the text array column (it gets toasted by postgres). For a consequent update of that row, instead of returning the toasted value placeholder, it throws error the above error.

      Do you see the same behaviour using the latest released Debezium version?

      Same issue in latest stable release (1.9.5.Final), yet to check in non stable releases

      Do you have the connector logs, ideally from start till finish?

      Attached

      How to reproduce the issue using our tutorial deployment?

      •  Create a table with multiple columns, including a column of type _text
      • Insert a row with a very large value for the _text column (it should get toasted)
      • Execute an update for that particular row, but for any other column other than the _text column
      • Debezium will throw the error, and will place a null value in the debezium message for the _text column, instead of the toasted value placeholder

      Attachments

        1. array_type_toast_fix.patch
          13 kB
          Pranav Hegde
        2. debezium-publisher.log
          29 kB
          Pranav Hegde

        Activity

          People

            jpechane Jiri Pechanec
            pranavh4 Pranav Hegde (Inactive)
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated: