Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-6573

Refactor errors.max.retries to common connector framework

      In order to make your issue reports as actionable as possible, please provide the following information, depending on the issue type.

      Bug report

      when mysql on aws(rds) is changed by blue/green update(binlog become invalid), mysql connector keep trying to connect mysql db. and it's status is running 

      the bin log is not valid, so the connector must be failed

       

      here is error log and it try to connect repeatedly

      [2023-06-19 06:59:55,690] ERROR Producer failure (io.debezium.pipeline.ErrorHandler)
      io.debezium.DebeziumException: Client requested master to start replication from position > file size Error code: 1236; SQLSTATE: HY000.
        at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.wrap(MySqlStreamingChangeEventSource.java:1236)
        at io.debezium.connector.mysql.MySqlStreamingChangeEventSource$ReaderThreadLifecycleListener.onCommunicationFailure(MySqlStreamingChangeEventSource.java:1281)
        at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:1079)
        at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:631)
        at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:932)
        at java.base/java.lang.Thread.run(Thread.java:829)
      Caused by: com.github.shyiko.mysql.binlog.network.ServerException: Client requested master to start replication from position > file size
        at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:1043)
        ... 3 more
      [2023-06-19 06:59:55,691] WARN Retry 1 of unlimited retries will be attempted (io.debezium.pipeline.ErrorHandler)
      [2023-06-19 06:59:55,692] INFO Stopped reading binlog after 0 events, no new offset was recorded (io.debezium.connector.mysql.MySqlStreamingChangeEventSource)
      [2023-06-19 06:59:55,969] WARN Going to restart connector after 10 sec. after a retriable exception (io.debezium.connector.common.BaseSourceTask)
      [2023-06-19 07:00:00,677] INFO Finished streaming (io.debezium.pipeline.ChangeEventSourceCoordinator)
      [2023-06-19 07:00:00,680] INFO Connection gracefully closed (io.debezium.jdbc.JdbcConnection)
      [2023-06-19 07:00:00,681] INFO [Producer clientId=_mysql.ver.up-schemahistory] Closing the Kafka producer with timeoutMillis = 30000 ms. (org.apache.kafka.clients.producer.KafkaProducer)
      [2023-06-19 07:00:00,684] INFO Metrics scheduler closed (org.apache.kafka.common.metrics.Metrics)
      [2023-06-19 07:00:00,684] INFO Closing reporter org.apache.kafka.common.metrics.JmxReporter (org.apache.kafka.common.metrics.Metrics)
      [2023-06-19 07:00:00,684] INFO Metrics reporters closed (org.apache.kafka.common.metrics.Metrics)
      [2023-06-19 07:00:00,684] INFO App info kafka.producer for _mysql.ver.up-schemahistory unregistered (org.apache.kafka.common.utils.AppInfoParser)
      [2023-06-19 07:00:00,684] WARN WorkerSourceTask{id=source.mysql.ver.up-0} failed to poll records from SourceTask. Will retry operation. (org.apache.kafka.connect.runtime.WorkerSourceTask)
      org.apache.kafka.connect.errors.RetriableException: An exception occurred in the change event producer. This connector will be restarted.
        at io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:68)
        at io.debezium.connector.mysql.MySqlStreamingChangeEventSource$ReaderThreadLifecycleListener.onCommunicationFailure(MySqlStreamingChangeEventSource.java:1281)
        at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:1079)
        at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:631)
        at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:932)
        at java.base/java.lang.Thread.run(Thread.java:829)
      Caused by: io.debezium.DebeziumException: Client requested master to start replication from position > file size Error code: 1236; SQLSTATE: HY000.
        at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.wrap(MySqlStreamingChangeEventSource.java:1236)
        ... 5 more
      Caused by: com.github.shyiko.mysql.binlog.network.ServerException: Client requested master to start replication from position > file size
        at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:1043)
        ... 3 more
      [2023-06-19 07:00:00,685] INFO Awaiting end of restart backoff period after a retriable error (io.debezium.connector.common.BaseSourceTask)
      [2023-06-19 07:00:02,685] INFO Awaiting end of restart backoff period after a retriable error (io.debezium.connector.common.BaseSourceTask)
      [2023-06-19 07:00:04,685] INFO Awaiting end of restart backoff period after a retriable error (io.debezium.connector.common.BaseSourceTask)
      [2023-06-19 07:00:06,686] INFO Awaiting end of restart backoff period after a retriable error (io.debezium.connector.common.BaseSourceTask)
      [2023-06-19 07:00:08,686] INFO Awaiting end of restart backoff period after a retriable error (io.debezium.connector.common.BaseSourceTask)

      What Debezium connector do you use and what version?

      2.2.1

      Implementation ideas (optional)

      add max retries to mysql connector config like sqlserver config

            [DBZ-6573] Refactor errors.max.retries to common connector framework

            Errata Tool added a comment -

            Since the problem described in this issue should be resolved in a recent advisory, it has been closed.

            For information on the advisory (Red Hat build of Debezium 2.3.4 release), and where to find the updated files, follow the link below.

            If the solution does not work for you, open a new bug report.
            https://access.redhat.com/errata/RHEA-2023:5471

            Errata Tool added a comment - Since the problem described in this issue should be resolved in a recent advisory, it has been closed. For information on the advisory (Red Hat build of Debezium 2.3.4 release), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2023:5471

            Released

            Debezium Builder added a comment - Released

            vjuranek@redhat.com 

            i make a pr : https://github.com/debezium/debezium/pull/4642

             

            but it failed, because of check style 

            in my local pc, checkstyel is success

             

            here is command, what i did  

            mvn -f debezium-core net.revelc.code.formatter:formatter-maven-plugin:2.20.0:format

            wornjs seo (Inactive) added a comment - vjuranek@redhat.com   i make a pr : https://github.com/debezium/debezium/pull/4642   but it failed, because of check style  in my local pc, checkstyel is success   here is command, what i did   mvn -f debezium-core net.revelc.code.formatter:formatter-maven-plugin:2.20.0:format

            jaegwon.seo sure, any PRs are more than welcome. Thanks!

            Vojtech Juranek added a comment - jaegwon.seo sure, any PRs are more than welcome. Thanks!

            thanks for checking 
            can i make this pr? 

            wornjs seo (Inactive) added a comment - thanks for checking  can i make this pr? 

            jaegwon.seo you are right, only SQL server have this option, others have unlimited retries. We should probably make this option general for all connector with the default to unlimited retries.

            Vojtech Juranek added a comment - jaegwon.seo you are right, only SQL server have this option, others have unlimited retries. We should probably make this option general for all connector with the default to unlimited retries.

            wornjs seo (Inactive) added a comment - - edited

            jpechane 
            do you means getMaxRetriesOnError method in CommonConnectorConfig class?

            it always retrun -1(infinite) and only overrided in sql server connector 

            https://github.com/debezium/debezium/blob/main/debezium-core/src/main/java/io/debezium/config/CommonConnectorConfig.java#L1129

            public int getMaxRetriesOnError()

            {    // Limited retries currently supported by SQL Server connector only    return ErrorHandler.RETRIES_UNLIMITED; }

            wornjs seo (Inactive) added a comment - - edited jpechane   do you means getMaxRetriesOnError method in CommonConnectorConfig class? it always retrun -1(infinite) and only overrided in sql server connector  https://github.com/debezium/debezium/blob/main/debezium-core/src/main/java/io/debezium/config/CommonConnectorConfig.java#L1129 public int getMaxRetriesOnError() {    // Limited retries currently supported by SQL Server connector only    return ErrorHandler.RETRIES_UNLIMITED; }

            jaegwon.seo Hi, I beleive this is already in place, this was added to the core library. Could you please try it?

            Jiri Pechanec added a comment - jaegwon.seo Hi, I beleive this is already in place, this was added to the core library. Could you please try it?

              Unassigned Unassigned
              jaegwon.seo wornjs seo (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: