Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-1071

Support Postgresql arbitrary-dimensional array

XMLWordPrintable

    • Icon: Feature Request Feature Request
    • Resolution: Duplicate
    • Icon: Minor Minor
    • None
    • 0.9.0.Beta2
    • postgresql-connector
    • None

      Current behavior

      Debezium Postgresql connector does not support arbitrary-dimensional array. For example,

      In case of two-dimensional array of INT
      Postgresql connector warns it and the result value is null.
      Avro schema

      {
        "name":"c_arr_int",
        "type":[
          "null",
          {
            "type":"array",
            "items":[
              "null",
              "int"
            ],
            "connect.parameters":{
              "__debezium.source.column.type":"_INT4",
              "__debezium.source.column.length":"10"
            }
          }
        ],
        "default":null
      }
      

      the warning message

      [stgresValueConverter] Unexpected value for JDBC type 4 and column c_arr_int-element int4(10, 0) DEFAULT VALUE NULL: class=Integer[]
      

      In case of two-dimensional array of TEXT
      Postgresql connector does not warn as follow and the result value seems to be casted string.
      Avro schema

      {
        "name":"c_arr_text",
        "type":[
          "null",
          {
            "type":"array",
            "items":[
              "null",
              "string"
            ],
            "connect.parameters":{
              "__debezium.source.column.type":"_TEXT",
              "__debezium.source.column.length":"2147483647"
            }
          }
        ],
        "default":null
      }
      

      result value

      ['[Ljava.lang.String;@3ce73358']
      

      Request

      1. Warning should be logged.
        At least, as long as Debezium does not support arbitrary-dimensional array, warning should be logged when it finds a two or more dimensional array of any data type. Moreover, if "include.unknown.datatypes" is enabled, raw binary representation of a column should be set as the result.
      2. Support arbitrary-dimensional array
        I guess the reason why Debezium does not support it is that both avro and protobuf can not define an arbitrary-dimensional array schema.
        My suggestion is to define such array schema as a linear(flatten) array with the dimension infomation.

              Unassigned Unassigned
              takaomag_jira Takao Magoori (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: