Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-7131

Provide ability to set the connection version for HTTP Sink

XMLWordPrintable

    • Icon: Enhancement Enhancement
    • Resolution: Unresolved
    • Icon: Major Major
    • 2.5-backlog
    • 2.4.0.Final
    • debezium-server
    • None
    • False
    • None
    • False

      What Debezium connector do you use and what version?

      I'm using debezium server version 2.4.0.final

      What is the connector configuration?

       

      debezium.source.connector.class=io.debezium.connector.postgresql.PostgresConnector

debezium.source.offset.storage.file.filename=data/offsets.dat
debezium.source.offset.flush.interval.ms=0

debezium.source.database.hostname=db_url
debezium.source.database.port=5432
debezium.source.database.user=db_user
debezium.source.database.password=db_pwd
debezium.source.database.dbname=db_name
debezium.source.database.sslmode=disable

debezium.source.plugin.name=pgoutput

debezium.source.topic.prefix=attom_data

debezium.source.table.include.list=dbo.property_registry

debezium.source.decimal.handling.mode=string

debezium.sink.type=http
debezium.sink.http.url=http://192.168.0.104:4001/message
debezium.sink.http.timeout.ms=600000
debezium.sink.http.retries=50
debezium.sink.http.retry.interval.ms=5000

debezium.format.value=json

quarkus.log.console.json=false
quarkus.http.port=8081

       

      What is the captured database version and mode of depoyment?

      (E.g. on-premises, with a specific cloud provider, etc.)

      Postgres 15 using AWS RDS

      What behaviour do you expect?

      Debezium will start sending events to the HTTP sink endpoint.

      What behaviour do you see?

      When debezium is starting, I will get the following error:

      2023-11-10 17:39:52,992 ERROR [io.deb.rel.RelationalSnapshotChangeEventSource] (debezium-postgresconnector-attom_data-change-event-source-coordinator) Error during snapshot: java.util.concurrent.ExecutionException: java.lang.InterruptedException: Interrupted while snapshotting table dbo.property_registry
    at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
    at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
    at io.debezium.relational.RelationalSnapshotChangeEventSource.createDataEvents(RelationalSnapshotChangeEventSource.java:463)
    at io.debezium.relational.RelationalSnapshotChangeEventSource.doExecute(RelationalSnapshotChangeEventSource.java:165)
    at io.debezium.pipeline.source.AbstractSnapshotChangeEventSource.execute(AbstractSnapshotChangeEventSource.java:92)
    at io.debezium.pipeline.ChangeEventSourceCoordinator.doSnapshot(ChangeEventSourceCoordinator.java:253)
    at io.debezium.pipeline.ChangeEventSourceCoordinator.doSnapshot(ChangeEventSourceCoordinator.java:237)
    at io.debezium.pipeline.ChangeEventSourceCoordinator.executeChangeEventSources(ChangeEventSourceCoordinator.java:189)
    at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:137)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.InterruptedException: Interrupted while snapshotting table dbo.property_registry
    at io.debezium.relational.RelationalSnapshotChangeEventSource.doCreateDataEventsForTable(RelationalSnapshotChangeEventSource.java:551)
    at io.debezium.relational.RelationalSnapshotChangeEventSource.lambda$createDataEventsForTableCallable$6(RelationalSnapshotChangeEventSource.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    ... 5 more2023-11-10 17:39:52,994 INFO  [io.deb.pip.sou.AbstractSnapshotChangeEventSource] (debezium-postgresconnector-attom_data-change-event-source-coordinator) Snapshot - Final stage
2023-11-10 17:39:52,995 WARN  [io.deb.pip.sou.AbstractSnapshotChangeEventSource] (debezium-postgresconnector-attom_data-change-event-source-coordinator) Snapshot was not completed successfully, it will be re-executed upon connector restart
2023-11-10 17:39:52,995 ERROR [io.deb.pip.ErrorHandler] (debezium-postgresconnector-attom_data-change-event-source-coordinator) Producer failure: io.debezium.DebeziumException: java.util.concurrent.ExecutionException: java.lang.InterruptedException: Interrupted while snapshotting table dbo.property_registry
    at io.debezium.pipeline.source.AbstractSnapshotChangeEventSource.execute(AbstractSnapshotChangeEventSource.java:101)
    at io.debezium.pipeline.ChangeEventSourceCoordinator.doSnapshot(ChangeEventSourceCoordinator.java:253)
    at io.debezium.pipeline.ChangeEventSourceCoordinator.doSnapshot(ChangeEventSourceCoordinator.java:237)
    at io.debezium.pipeline.ChangeEventSourceCoordinator.executeChangeEventSources(ChangeEventSourceCoordinator.java:189)
    at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:137)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.util.concurrent.ExecutionException: java.lang.InterruptedException: Interrupted while snapshotting table dbo.property_registry
    at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
    at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
    at io.debezium.relational.RelationalSnapshotChangeEventSource.createDataEvents(RelationalSnapshotChangeEventSource.java:463)
    at io.debezium.relational.RelationalSnapshotChangeEventSource.doExecute(RelationalSnapshotChangeEventSource.java:165)
    at io.debezium.pipeline.source.AbstractSnapshotChangeEventSource.execute(AbstractSnapshotChangeEventSource.java:92)
    ... 9 more
Caused by: java.lang.InterruptedException: Interrupted while snapshotting table dbo.property_registry
    at io.debezium.relational.RelationalSnapshotChangeEventSource.doCreateDataEventsForTable(RelationalSnapshotChangeEventSource.java:551)
    at io.debezium.relational.RelationalSnapshotChangeEventSource.lambda$createDataEventsForTableCallable$6(RelationalSnapshotChangeEventSource.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    ... 5 more2023-11-10 17:39:52,995 INFO  [io.deb.pip.ChangeEventSourceCoordinator] (debezium-postgresconnector-attom_data-change-event-source-coordinator) Connected metrics set to 'false'
2023-11-10 17:39:52,997 INFO  [io.deb.pip.sig.SignalProcessor] (pool-7-thread-1) SignalProcessor stopped
2023-11-10 17:39:53,015 INFO  [io.deb.jdb.JdbcConnection] (pool-11-thread-1) Connection gracefully closed
2023-11-10 17:39:53,018 INFO  [org.apa.kaf.con.sto.FileOffsetBackingStore] (pool-7-thread-1) Stopped FileOffsetBackingStore
2023-11-10 17:39:53,018 ERROR [io.deb.ser.ConnectorLifecycle] (pool-7-thread-1) Connector completed: success = 'false', message = 'Stopping connector after error in the application's handler method: java.io.IOException', error = 'java.lang.InterruptedException: java.io.IOException': java.lang.InterruptedException: java.io.IOException
    at io.debezium.server.http.HttpChangeConsumer.recordSent(HttpChangeConsumer.java:197)
    at io.debezium.server.http.HttpChangeConsumer.handleBatch(HttpChangeConsumer.java:164)
    at io.debezium.embedded.ConvertingEngineBuilder.lambda$notifying$2(ConvertingEngineBuilder.java:101)
    at io.debezium.embedded.EmbeddedEngine.pollRecords(EmbeddedEngine.java:1008)
    at io.debezium.embedded.EmbeddedEngine.run(EmbeddedEngine.java:755)
    at io.debezium.embedded.ConvertingEngineBuilder$2.run(ConvertingEngineBuilder.java:229)
    at io.debezium.server.DebeziumServer.lambda$start$1(DebeziumServer.java:170)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)

      I believe the most important part is this line:

      at io.debezium.server.http.HttpChangeConsumer.recordSent(HttpChangeConsumer.java:197)

      Also, I have a very similar config but using rabbitmq sink and that one works fine. I also can access the endpoint using `curl` just fine.

      Do you see the same behaviour using the latest relesead Debezium version?

      (Ideally, also verify with latest Alpha/Beta/CR version)

      I'm already using the latest version (2.4.0.final), but I also tried 2.2.1 before and it has the same error.

      Do you have the connector logs, ideally from start till finish?

      (You might be asked later to provide DEBUG/TRACE level log)

      I couldn't attach the full log here, but if needed I can upload somewhere and put the link here

      How to reproduce the issue using our tutorial deployment?

      • What I did was download debezium server 2.4.0;
      • Put the above config;
      • Create the data directory with `mkdir data`;
      • Start the server with `./run.sh`.

       

        1. debezium-1.log
          74 kB

            Unassigned Unassigned
            sezaru Eduardo Alexandre (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: