Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-1400

Debezium does not get the first change after creating the replication slot in PostgreSQL

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Critical Critical
    • 0.10.0.Beta3
    • 0.9.5.Final, 0.10.0.Beta2
    • postgresql-connector
    • None
    • Hide

      1 - Run official containers:
      docker run -it -d --name kafka_zookeeper -p 2181:2181 -p 2888:2888 -p 3888:3888 debezium/zookeeper
      docker run -it -d --name kafka -p 9092:9092 --link kafka_zookeeper:zookeeper debezium/kafka
      docker run -it -d --env-file env --name kafka_connect -p 8083:8083 -e GROUP_ID=1 -e CONFIG_STORAGE_TOPIC=my-connect-configs -e OFFSET_STORAGE_TOPIC=my-connect-offsets -e ADVERTISED_HOST_NAME=$(echo $DOCKER_HOST | cut -f3 -d’/’ | cut -f1 -d’:’) --link kafka_zookeeper:zookeeper --link kafka:kafka debezium/connect

      2 - Create connector using kafka_config.json (attached to this issue):

      curl -X POST -H "Content-Type: application/json" --data @kafka_config.json localhost:8083/connectors | jq -r

      3 - Perform any UPDATE or INSERT query on a watched table.
      Example: insert into public.fake (id) values(1);

      Show
      1 - Run official containers: docker run -it -d --name kafka_zookeeper -p 2181:2181 -p 2888:2888 -p 3888:3888 debezium/zookeeper docker run -it -d --name kafka -p 9092:9092 --link kafka_zookeeper:zookeeper debezium/kafka docker run -it -d --env-file env --name kafka_connect -p 8083:8083 -e GROUP_ID=1 -e CONFIG_STORAGE_TOPIC=my-connect-configs -e OFFSET_STORAGE_TOPIC=my-connect-offsets -e ADVERTISED_HOST_NAME=$(echo $DOCKER_HOST | cut -f3 -d’/’ | cut -f1 -d’:’) --link kafka_zookeeper:zookeeper --link kafka:kafka debezium/connect 2 - Create connector using kafka_config.json (attached to this issue): curl -X POST -H "Content-Type: application/json" --data @kafka_config.json localhost:8083/connectors | jq -r 3 - Perform any UPDATE or INSERT query on a watched table. Example: insert into public.fake (id) values(1);

      Debezium does not get the first change after creating the replication slot in PostgreSQL. The problem happens using:

      *wal2json
      wal2json_rds
      wal2json_rds_streaming*

      But only in wal2json_rds_streaming, we have a log message of level TRACE saying that you received an empty message. When executing a SELECT in the replication slot, we see that the change has been registered normally and is not empty (postgres_peek_changes.png).

      We use Postgres 11 in AWS RDS.
      But we also tested with a local postgres https://hub.docker.com/r/debezium/postgres and the problem still occurred. The subsequent changes after the first one are normally caught by the connector. Only the first one misses.

      LOG:

      _2019-07-16 15:36:39,386 TRACE Postgres|postgres-dev|records-stream-producer Chunk arrived from database ]
      } [io.debezium.connector.postgresql.connection.wal2json.StreamingWal2JsonMessageDecoder]
      2019-07-16 15:36:39,386 WARN Postgres|postgres-dev|records-stream-producer Got out of order chunk ]
      } , recording artifical TX [io.debezium.connector.postgresql.connection.wal2json.StreamingWal2JsonMessageDecoder]
      2019-07-16 15:36:39,387 TRACE Postgres|postgres-dev|records-stream-producer Empty change arrived [io.debezium.connector.postgresql.connection.wal2json.StreamingWal2JsonMessageDecoder]
      2019-07-16 15:36:39,387 TRACE Postgres|postgres-dev|records-stream-producer Received empty message [io.debezium.connector.postgresql.RecordsStreamProducer]
      _

        1. debezium_connector.log
          28 kB
        2. kafka_config.json
          0.7 kB
        3. postgres_peek_changes.png
          postgres_peek_changes.png
          69 kB

              jpechane Jiri Pechanec
              alexsanderpereira Alexsander Pereira (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: