Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-3388

Failed to run multiple debezium MySQL Connectors to work with multitenant. i.e, when multiple source connectors connected to different databases with in the same database server in azure.

    XMLWordPrintable

Details

    • False
    • False
    • Undefined
    • Hide

      The following are the connectors configurations.

      Show
      The following are the connectors configurations.

    Description

      We tried to connect multiple debezium MySQL connectors with different database with in the same azure mysql  database server for different clients and it was running with two brokers on single node. For one connector it was working fine. Second source connector worked  for few minutes and returned the following errors.

       -------------

      {"name":"Source-connector2","connector":\{"state":"RUNNING","worker_id":"10.0.0.9:8083"}

      ,"tasks":[\{"id":0,"state":"FAILED","worker_id":"10.0.0.9:8083","trace":"org.apache.kafka.connect.errors.ConnectException: An exception occurred in the change event producer. This connector will be stopped.\n\tat io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:42)\n\tat io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:121)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:748)\nCaused by: io.debezium.DebeziumException: java.lang.RuntimeException: java.sql.SQLNonTransientConnectionException: Communications link failure during rollback(). Transaction resolution unknown.\n\tat io.debezium.pipeline.source.AbstractSnapshotChangeEventSource.execute(AbstractSnapshotChangeEventSource.java:82)\n\tat io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:105)\n\t... 5 more\nCaused by: java.lang.RuntimeException: java.sql.SQLNonTransientConnectionException: Communications link failure during rollback(). Transaction resolution unknown.\n\tat io.debezium.relational.RelationalSnapshotChangeEventSource.rollbackTransaction(RelationalSnapshotChangeEventSource.java:521)\n\tat io.debezium.relational.RelationalSnapshotChangeEventSource.doExecute(RelationalSnapshotChangeEventSource.java:159)\n\tat io.debezium.pipeline.source.AbstractSnapshotChangeEventSource.execute(AbstractSnapshotChangeEventSource.java:71)\n\t... 6 more\nCaused by: java.sql.SQLNonTransientConnectionException: Communications link failure during rollback(). Transaction resolution unknown.\n\tat com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:110)\n\tat com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)\n\tat com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:89)\n\tat com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:63)\n\tat com.mysql.cj.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:1853)\n\tat io.debezium.relational.RelationalSnapshotChangeEventSource.rollbackTransaction(RelationalSnapshotChangeEventSource.java:518)\n\t... 8 more\n"}],"type":"source"}
       
      and  after some time 
      [2021-04-08 03:46:43,775] ERROR Error during binlog processing. Last offset stored = null, binlog reader near position = mysql-bin.000768/29360715 (io.debezium.connector.mysql.MySqlStreamingChangeEventSource:1125)
      [2021-04-08 03:46:43,776] ERROR Producer failure (io.debezium.pipeline.ErrorHandler:31)
      io.debezium.DebeziumException: A slave with the same server_uuid/server_id as this slave has connected to the master; the first event 'mysql-bin.000749' at 87372313, the last event read from 'c:\work\binlogs\mysql-bin.000768' at 29360715, the last byte read from 'c:\work\binlogs\mysql-bin.000768' at 29360715. Error code: 1236; SQLSTATE: HY000.
      at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.wrap(MySqlStreamingChangeEventSource.java:1225)
      at io.debezium.connector.mysql.MySqlStreamingChangeEventSource$ReaderThreadLifecycleListener.onCommunicationFailure(MySqlStreamingChangeEventSource.java:1264)
      at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:962)
      at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:595)
      at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:839)
      at java.lang.Thread.run(Thread.java:748)
      Caused by: com.github.shyiko.mysql.binlog.network.ServerException: A slave with the same server_uuid/server_id as this slave has connected to the master; the first event 'mysql-bin.000749' at 87372313, the last event read from 'c:\work\binlogs\mysql-bin.000768' at 29360715, the last byte read from 'c:\work\binlogs\mysql-bin.000768' at 29360715.
      at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:926)
      ... 3 more
      [2021-04-08 03:46:43,777] INFO Stopped reading binlog after 0 events, no new offset was recorded (io.debezium.connector.mysql.MySqlStreamingChangeEventSource:1238)
      [2021-04-08 03:46:44,210] INFO WorkerSourceTask{id=Source-connector1-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:478)
      [2021-04-08 03:46:44,212] INFO WorkerSourceTask{id=Source-connector1-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:495)
      [2021-04-08 03:46:44,212] ERROR WorkerSourceTask{id=Source-connector1-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask:187)
      org.apache.kafka.connect.errors.ConnectException: An exception occurred in the change event producer. This connector will be stopped.
      at io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:42)
      at io.debezium.connector.mysql.MySqlStreamingChangeEventSource$ReaderThreadLifecycleListener.onCommunicationFailure(MySqlStreamingChangeEventSource.java:1264)
      at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:962)
      at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:595)
      at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:839)
      at java.lang.Thread.run(Thread.java:748)
      Caused by: io.debezium.DebeziumException: A slave with the same server_uuid/server_id as this slave has connected to the master; the first event 'mysql-bin.000749' at 87372313, the last event read from 'c:\work\binlogs\mysql-bin.000768' at 29360715, the last byte read from 'c:\work\binlogs\mysql-bin.000768' at 29360715. Error code: 1236; SQLSTATE: HY000.
      at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.wrap(MySqlStreamingChangeEventSource.java:1225)
      ... 5 more
      Caused by: com.github.shyiko.mysql.binlog.network.ServerException: A slave with the same server_uuid/server_id as this slave has connected to the master; the first event 'mysql-bin.000749' at 87372313, the last event read from 'c:\work\binlogs\mysql-bin.000768' at 29360715, the last byte read from 'c:\work\binlogs\mysql-bin.000768' at 29360715.
      at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:926)
      ... 3 more
      [2021-04-08 03:46:44,215] INFO Stopping down connector (io.debezium.connector.common.BaseSourceTask:192)
      [2021-04-08 03:46:44,215] INFO Finished streaming (io.debezium.pipeline.ChangeEventSourceCoordinator:141)
      [2021-04-08 03:46:44,641] INFO Connection gracefully closed (io.debezium.jdbc.JdbcConnection:946)

      Attachments

        1. jdbc_sink1.json
          1 kB
        2. jdbc_sink2.json
          1 kB
        3. sourceconnector1.json
          1 kB
        4. sourceconnector2.json
          1 kB

        Activity

          People

            Unassigned Unassigned
            sandhyarani3103 Sandhya Rani (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: