Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-4178

Improve documentation on handling EventDataDeserializationExceptions

    XMLWordPrintable

Details

    • Enhancement
    • Resolution: Unresolved
    • Major
    • 1.9-backlog
    • None
    • mysql-connector
    • None
    • False
    • False

    Description

      The current F.A.Q. article suggests changing MySQL configuration as follows:

      set global slave_net_timeout = 120; (default was 30sec)
      set global thread_pool_idle_timeout = 120;
      

      This may be misleading since as far as I can tell, thread_pool_idle_timeout doesn’t exist in MySQL (it’s probably only available in MariaDB): 

      Server version: 8.0.26 Homebrew

      mysql> select @@global.thread_pool_idle_timeout;
      ERROR 1193 (HY000): Unknown system variable 'thread_pool_idle_timeout'

      As for slave_net_timeout, according to the documentation (also, this article), it’s applied to a MySQL replica, so it shouldn’t affect either of the MySQL primary node and Debezium.

      Regardless of whether the above approach works, the article suggests a solution without describing the actual problem.

      In our case, the root cause was the insufficient heap size on the Connect worker node that made JVM use a significant part of the CPU resources on GC and not communicate with the database. See the details in the comments.

      As discussed offline, the following recommendations could be made:

      1. Monitor the JMX metrics in general. If there are scraping failures (like it was in our case), it's likely that there is CPU starvation that may be causing this issue.
      2. Monitor the time that JVM spends in GC.
      3. Monitor the CPU usage in general. If it's constantly high regardless of the actual rate of incoming events (like it was in our case), it's likely that the CPU is used primarily for GC, not the actual work.

      Attachments

        Activity

          People

            Unassigned Unassigned
            sergeimorozov Sergei Morozov
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: