Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-4130

Explore columnar design for metamodel

    XMLWordPrintable

Details

    • Task
    • Resolution: Unresolved
    • Major
    • 1.9-backlog
    • None
    • core-library
    • None
    • False
    • False

    Description

      When capturing large databases with tens or hundreds of thousands of tables (as e.g. the case for multi-tenancy use cases frequently), the Debezium connectors will use a significant amount of memory to store the table metadata model.

      This could likely be improved by abandoning the current design with a ColumnImpl object per table column in favor of a columnar design of TableImpl which stores the information per attribute type in arrays:

      private String[] columnNames;
      private int [] columnLengths;
      private boolean[] columnNullabilities;
      ...
      

      Access would be by index on the Table API:

      int length = table.getLength(myColumnIndex);
      

      Column indexes should be obtained once per incoming event ideally, avoiding repeated look-ups.

      Validation of the approach could be done via JfrUnit by comparing/asserting the object allocations for a given large model. This would also prevent future regressions leading to increased memory usage again.

      Attachments

        Activity

          People

            Unassigned Unassigned
            gunnar.morling Gunnar Morling
            Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: