Details

      Description

      Currently we are working on updating our production AWS Aurora Mysql DB's to enable GTID's using debezium v1.2.0. In order to maintain high availability for our platform and our services powered by debezium we were attempting to do the following

      1. Spin up a second primary DB from a snapshot backup of our current primary
      2. Configure the new primary for GTID_MODE=ON_PERMISSIVE
      3. Do primary to primary replication to from the current to the new DB
      4. Start a new debezium connector and re-snapshot / source data from the new primary's binlog
      5. Switching over read traffic to the new primary DB
      6. Switch over write traffic to the new primary DB after all read traffic is switched over
        We are attempting the above strategy so we can switch read traffic on our system over to the new primary before we cutover write traffic in order to avoid any downtime on reads since turning on GTID requires a restart of the full Aurora cluster. When we attempt the above steps in our test environment we see the following stacktrace being thrown and connector dying {{}}
         org.apache.kafka.connect.errors.ConnectException: The replication sender thread cannot start in AUTO_POSITION mode: this server has GTID_MODE = ON_PERMISSIVE instead of ON. Error code: 1236; SQLSTATE: HY000. at io.debezium.connector.mysql.AbstractReader.wrap(AbstractReader.java:230) at io.debezium.connector.mysql.AbstractReader.failed(AbstractReader.java:196) at io.debezium.connector.mysql.BinlogReader$ReaderThreadLifecycleListener.onCommunicationFailure(BinlogReader.java:1061) at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:950) at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:580) at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:825) at java.lang.Thread.run(Thread.java:748) Caused by: com.github.shyiko.mysql.binlog.network.ServerException: The replication sender thread cannot start in AUTO_POSITION mode: this server has GTID_MODE = ON_PERMISSIVE instead of ON. at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:914) ... 3 more

      After a bunch of testing it appears debezium only seems to currently support GTID_MODE=ON or GTID_MODE=OFF. Looking for help on the following questions
      1. Do you guys see this as a feasible approach?

      2. Is this just debezium lacking support for other GTID_MODE's, or possibly a bug in the way debezium is conditionally checking for GTID set? Looking at the source check for using GTID it looks like it defaults to using GTID if GTID_MODE is not explicitly OFF
      https://github.com/debezium/debezium/blob/master/debezium-connector-mysql/src/main/java/io/debezium/connector/mysql/MySqlJdbcContext.java#L190

      3. If question 2 is not a bug or lack of support and is performing this way for some technical reason, would exposing a configuration option for forcing debezium to ignore GTID's something that you guys would be interested in?
      https://github.com/debezium/debezium/blob/master/debezium-connector-mysql/src/main/java/io/debezium/connector/mysql/BinlogReader.java#L355

      Would be happy to possibly help contribute to question 2 or 3 depending on your thoughts

        Gliffy Diagrams

          Attachments

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                eric-weaver Eric Weaver
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated: