Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-513

AWS RDS Postgresql 9.6.5 not supporting "include-not-null" = "true" in connector setup

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Minor
    • 0.7.1
    • 0.7.0
    • postgresql-connector
    • None
    • Hide

      On RDS enabled logical_replication to 1 and reboot happened.
      After zookeeper, kafka and connect docker started from 0.7.0 tagged images connector configuration pushed with below config:

      curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" localhost:8083/connectors/ -d '{
      "name": "postgresql-connector",
      "config":

      { "connector.class": "io.debezium.connector.postgresql.PostgresConnector", "database.hostname": "<DB-IP-ADDRESS>", "database.port": "5432", "database.user": "<username>", "database.password": "<password>", "database.dbname": "testdb", "database.sslmode": "require", "plugin.name": "wal2json", "database.server.name": "dbserver1", "table.whitelist": "public.table1,public.table2", "column.blacklist": "public.table1.col1,public.table2.col1" }

      }'

      Getting below error on connect docker console.

      Show
      On RDS enabled logical_replication to 1 and reboot happened. After zookeeper, kafka and connect docker started from 0.7.0 tagged images connector configuration pushed with below config: curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" localhost:8083/connectors/ -d '{ "name": "postgresql-connector", "config": { "connector.class": "io.debezium.connector.postgresql.PostgresConnector", "database.hostname": "<DB-IP-ADDRESS>", "database.port": "5432", "database.user": "<username>", "database.password": "<password>", "database.dbname": "testdb", "database.sslmode": "require", "plugin.name": "wal2json", "database.server.name": "dbserver1", "table.whitelist": "public.table1,public.table2", "column.blacklist": "public.table1.col1,public.table2.col1" } }' Getting below error on connect docker console.

    Description

      2017-12-17 09:23:37,084 ERROR Postgres|dbserver1|records-stream-producer unexpected exception while streaming logical changes [io.debezium.connector.postgresql.RecordsStreamProducer]
      org.postgresql.util.PSQLException: ERROR: option "include-not-null" = "true" is unknown
      Where: slot "debezium", output plugin "wal2json", in the startup callback
      at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2412)
      at org.postgresql.core.v3.QueryExecutorImpl.processCopyResults(QueryExecutorImpl.java:1043)
      at org.postgresql.core.v3.QueryExecutorImpl.readFromCopy(QueryExecutorImpl.java:962)
      at org.postgresql.core.v3.CopyDualImpl.readFromCopy(CopyDualImpl.java:41)
      at org.postgresql.core.v3.replication.V3PGReplicationStream.receiveNextData(V3PGReplicationStream.java:145)
      at org.postgresql.core.v3.replication.V3PGReplicationStream.readInternal(V3PGReplicationStream.java:114)
      at org.postgresql.core.v3.replication.V3PGReplicationStream.read(V3PGReplicationStream.java:60)
      at io.debezium.connector.postgresql.connection.PostgresReplicationConnection$1.read(PostgresReplicationConnection.java:179)
      at io.debezium.connector.postgresql.RecordsStreamProducer.streamChanges(RecordsStreamProducer.java:119)
      at io.debezium.connector.postgresql.RecordsStreamProducer.lambda$start$1(RecordsStreamProducer.java:105)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)
      2017-12-17 09:23:37,484 INFO || WorkerSourceTask

      {id=postgresql-connector-0} Committing offsets [org.apache.kafka.connect.runtime.WorkerSourceTask]
      2017-12-17 09:23:37,486 INFO || WorkerSourceTask{id=postgresql-connector-0}

      flushing 0 outstanding messages for offset commit [org.apache.kafka.connect.runtime.WorkerSourceTask]

      Attachments

        Issue Links

          is related to

          Enhancement - An enhancement to or refactoring of existing functionality that is not configurable by an end user (typically a change made by an internal team that affects users) DBZ-517 Provide a wal2json plug-in mode enforcing RDS environment

          • Critical - To be worked on immediately following BLOCKER issues.
          • Closed

          Activity

            People

              Unassigned Unassigned
              surajsavita Suraj Savita (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: