Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-5126

Memory leak in EventDeserializer caused by tableMapEventByTableId

    XMLWordPrintable

Details

    • False
    • None
    • False

    Description

      Feature request or enhancement

      Memory leak in EventDeserializer caused by tableMapEventByTableId

      Related:

      https://github.com/osheroff/mysql-binlog-connector-java/issues/62

      https://github.com/osheroff/mysql-binlog-connector-java/pull/2/files

       

      Table id is not a constant value in one MySQL instance, it will assign a new id when there is no cached table id found, or roll to a new id when the cache in MySQL server `my_hash_search_using_hash_value` {}is full. Thus, it is not necessary keeping this tableMapEventByTableId map all the time.

       

      MySqlStreamingChangeEventSource:224

      // Set up the event deserializer with additional type(s) ...
      final Map<Long, TableMapEventData> tableMapEventByTableId = new HashMap<Long, TableMapEventData>(); 

      We override the EventDeserializer and use HashMap to store the TableMapData here.

      Assume that the MySQL server's `table_def_cache` size is 400 and we have over 400 tables keep writing, the same table's table id may be increased over time and the tableMapEventByTableId will become unbounded.

      Implementation ideas (optional)

      Clear it on ROTATE event

      Or use LRUCache instead of HashMap (Not safe in extreme cases)

      Attachments

        Activity

          People

            Unassigned Unassigned
            jiabao-sun Jiabao Sun (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: