Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-682

Fix high memory consumption for database per customer use case

    XMLWordPrintable

Details

    • Feature Request
    • Resolution: Duplicate
    • Major
    • None
    • None
    • None
    • None
    • 0
    • 0% 0%

    Description

      Problem:
      If you have data structured in a way that each of your client business has their own database and you optimize MySQL server utilization by packing many clients on the same server, then the total schema size on one Mysql node can get pretty large. A large schema, in turn, makes connecting debezium problematic because it will need a large amount of memory just to store the full schema in memory. Imagine demo inventory database, but stored 1000+ times under different database names.

      Some numbers:
      One server can contain ~1000 databases, each which have 95% similar structure. One database contains around 100 tables with the total of ~1200 columns. One entire MySQL server, therefore, contains ca 1.2M columns.

      From FAQ I see the Debezium default 256mb instance can safely handle 10 000 column schema.
      I can tell that given 2GB of memory the 1.2M columns also work, but it becomes very expensive if you have tens of servers.

      Therefore, my feature request:
      When storing table schemas in memory, could debezium run it through deduplication layer, so only unique tables get stored, copies just referenced to the first one.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              pimpelsang Eero Koplimets (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: