Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-6439

Snapshot step 5 - Reading structure of captured tables time too long

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Major
    • 2.3.0.Beta1
    • 2.2.0.Final
    • oracle-connector
    • None

    Description

      What Debezium connector do you use and what version?

      debezium-connector-oracle-2.2.0.Final

      What is the connector configuration?

      { "connector.class": "io.debezium.connector.oracle.OracleConnector", "snapshot.locking.mode": "none", "tasks.max": "1", "schema.include.list": "TEST", "log.mining.strategy": "online_catalog", "schema.history.internal.store.only.captured.tables.ddl": "true", "topic.prefix": "TEST", "decimal.handling.mode": "double", "schema.history.internal.kafka.topic": "DBHISTORY. TEST01", "signal.data.collection": "ORCL. TEST. DEBEZIUM_SIGNAL", "log.mining.archive.log.only.mode": "false", "log.mining.archive.log.hours": "0", "log.mining.transaction.retention.hours": "12", "database.user": "user", "database.dbname": "ORCL", "time.precision.mode": "connect", "schema.history.internal.kafka.bootstrap.servers": "broker:9092", "event.processing.failure.handling.mode": "warn", "schema.history.internal.skip.unparseable.ddl": "true", "database.port": "1521", "database.serverTimezone": "UTC", " database.hostname": "172.0.1.1", "database.password": "password", "table.include.list": "TEST. TBTEST,TEST. DEBEZIUM_SIGNAL", "snapshot.mode": "initial" }

      What behavior do you expect?

      In the fifth step of the snapshot, if you have a specified table, you do not need to get all the table metadata to save snapshot time.

      What behavior do you see?

      I have over 30000,100 tables in my database, and I only need to capture more than <> tables. During the snapshot, I found out that it took more than an hour. It's been too long.

      At the same time, I examined the code in version 2.2 and found it in the OracleSnapshotChangeEventSource class, the readTableStructure method, and jdbcConnection. readSchema, passed the TableFilter parameter is empty. In the jdbcConnection class, readSchema method, if (tableFilter==null | | tableFilter. isInclusion (tableId)) {" was not filtered, resulting in a long time.

      Do you have connector logs, preferably from start to finish?

      (You may be asked to provide debug/trace level logs later)

      [2023-05-05 17:57:18,472] INFO Snapshot step 5 - Reading structure of captured tables (io.debezium.relational.RelationalSnapshotChangeEventSource)
      .....
      [2023-05-05 19:00:39,289] INFO Snapshot step 5.a - Creating connection pool (io.debezium.relational.RelationalSnapshotChangeEventSource)

       

      Attachments

        Issue Links

          Activity

            People

              vjuranek@redhat.com Vojtech Juranek
              sowly_wanglian@126.com wang lian (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: