Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-5828

PubSub sink does not ignore tombstone messages

    XMLWordPrintable

Details

    • Bug
    • Resolution: Unresolved
    • Major
    • 2.4-backlog
    • 2.0.0.Final
    • debezium-server
    • None
    • False
    • None
    • False

    Description

      What Debezium connector do you use and what version?

      Debezium Server 2.0

      What is the connector configuration?

      debezium.sink.type=pubsub
      debezium.sink.pubsub.project.id=handshake-development-204119
      debezium.sink.pubsub.address=pubsub:8085
      debezium.source.connector.class=io.debezium.connector.postgresql.PostgresConnector
      debezium.source.topic.prefix=handshake
      debezium.source.offset.storage.file.filename=data/offsets.dat
      debezium.source.database.hostname=pg13database
      debezium.source.database.port=5432
      debezium.source.database.user=postgres
      debezium.source.database.password=
      debezium.source.database.dbname=handshake_dev
      debezium.source.schema.include.list=public
      debezium.source.plugin.name=pgoutput
      debezium.transforms=singleTopic
      debezium.transforms.singleTopic.type=io.debezium.transforms.ByLogicalTableRouter
      debezium.transforms.singleTopic.topic.regex=(.*)
      debezium.transforms.singleTopic.topic.replacement=debezium-record-changes
      debezium.transforms.singleTopic.key.enforce.uniqueness=false 

      What is the captured database version and mode of depoyment?

      Dockerized PG13 instance

      What behaviour do you expect?

      The server processes the delete message without issue.

      What behaviour do you see?

      The server throws an InvalidArgument:

      java.util.concurrent.ExecutionException: com.google.api.gax.rpc.InvalidArgumentException: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: Some messages are empty

      I've traced this to there being a second EmbeddedEngineChangeEvent with a null value. I suspect it might align with RELATION message from PG as I see the expected delete message first with "op": "d". I've included a screen shot from my IntelliJ remote debug set up.

       

      Do you see the same behaviour using the latest relesead Debezium version?

      Yes, this is the latest released version

      Do you have the connector logs, ideally from start till finish?

      Yes

      How to reproduce the issue using our tutorial deployment?

      Not sure. I am using local docker containers for PG13 and the Google Pubsub Emulator.

      Implementation ideas (optional)

      A quick fix would be to ignore EmbeddedEngineChangeEvent with null values in PubSubChangeConsumer. Though it might be better to understand why this null event is being generated in the first place.

      Attachments

        Activity

          People

            Unassigned Unassigned
            smcauliffe43 Sean McAuliffe (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: