-
Feature Request
-
Resolution: Duplicate
-
Major
-
None
-
None
-
None
-
None
Problem:
If you have data structured in a way that each of your client business has their own database and you optimize MySQL server utilization by packing many clients on the same server, then the total schema size on one Mysql node can get pretty large. A large schema, in turn, makes connecting debezium problematic because it will need a large amount of memory just to store the full schema in memory. Imagine demo inventory database, but stored 1000+ times under different database names.
Some numbers:
One server can contain ~1000 databases, each which have 95% similar structure. One database contains around 100 tables with the total of ~1200 columns. One entire MySQL server, therefore, contains ca 1.2M columns.
From FAQ I see the Debezium default 256mb instance can safely handle 10 000 column schema.
I can tell that given 2GB of memory the 1.2M columns also work, but it becomes very expensive if you have tens of servers.
Therefore, my feature request:
When storing table schemas in memory, could debezium run it through deduplication layer, so only unique tables get stored, copies just referenced to the first one.
- is duplicated by
-
DBZ-1104 High memory consumption in case of multi-tenant use cases
- Open