Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-3955

Debezium long start up times with large number of postgres schemas

XMLWordPrintable

    • False
    • False
    • Hide

      See attached for reproduction based on debezium-examples. 

      Note: db init takes about a minute to run before db is ready.

       

      docker-compose -f docker-compose-postgres.yaml up --build
curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" http://localhost:8083/connectors/ -d @register-postgres.json
      Show
      See attached for reproduction based on debezium-examples.  Note: db init takes about a minute to run before db is ready.   docker-compose -f docker-compose-postgres.yaml up --build curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" http: //localhost:8083/connectors/ -d @register-postgres.json

      With a large number of schemas (1k of 50tables each) Debezium can take an excessively long time to start up while waiting for metadata from the DB. This was tested on postgres with the latest 1.6 and 1.7 dbz releases. The behaviour seen is compounded when the db has performance issues, as the meta data query is being run for each schema and table rather than getting everything in a single query. eg, in an online production environment this startup could take a couple days or more.

      Primary Issues

      • metadata query is different and slower when table.exclude.list is included.
      • metadata is retrieved multiple times during startup/snapshot
      • metadata query is executing for each table rather than all at once or with schema filter in query.

      To see difference, compare time with table.exclude.list and without.

       

       

        1. Dockerfile
          0.1 kB
          Marcus English
        2. docker-compose-postgres.yaml
          0.9 kB
          Marcus English
        3. init.sql
          0.3 kB
          Marcus English
        4. register-postgres.json
          0.9 kB
          Marcus English

              Unassigned Unassigned
              marcus.english Marcus English (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Created:
                Updated: