Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-1311

Reading structure of captured tables is too slow

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Obsolete
    • Icon: Major Major
    • None
    • 0.9.5.Final
    • oracle-connector
    • None

      Hi,

      I have 3594 tables in my database schema:

      SELECT count(*)
FROM all_tables
WHERE owner='my_schema';
      COUNT(*)
      3594

      and I want to stream only one table.
      Snapshot step 5 takes several hours and falls after connector rebalancing:

      [2019-05-29T10:17:43,509][INFO][category=io.debezium.relational.HistorizedRelationalSnapshotChangeEventSource] Snapshot step 5 - Reading structure of captured tables
...
[2019-05-29T12:23:43,016][ERROR][category=io.debezium.pipeline.ErrorHandler] Producer failure
java.lang.RuntimeException: java.sql.SQLRecoverableException: Closed Connection
	at io.debezium.relational.HistorizedRelationalSnapshotChangeEventSource.rollbackTransaction(HistorizedRelationalSnapshotChangeEventSource.java:407)
	at io.debezium.relational.HistorizedRelationalSnapshotChangeEventSource.execute(HistorizedRelationalSnapshotChangeEventSource.java:172)
	at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:80)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.sql.SQLRecoverableException: Closed Connection
	at oracle.jdbc.driver.PhysicalConnection.getAutoCommit(PhysicalConnection.java:1828)
	at oracle.jdbc.driver.PhysicalConnection.rollback(PhysicalConnection.java:1953)
	at io.debezium.relational.HistorizedRelationalSnapshotChangeEventSource.rollbackTransaction(HistorizedRelationalSnapshotChangeEventSource.java:404)
	... 7 more

      Thread dump:

        java.lang.Thread.State: RUNNABLE
	  at sun.nio.ch.SocketDispatcher.read0(SocketDispatcher.java:-1)
	  at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43)
	  at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:276)
	  at sun.nio.ch.IOUtil.read(IOUtil.java:245)
	  at sun.nio.ch.IOUtil.read(IOUtil.java:223)
	  at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:355)
	  at oracle.net.nt.TimeoutSocketChannel.read(TimeoutSocketChannel.java:144)
	  at oracle.net.ns.NIOHeader.readHeaderBuffer(NIOHeader.java:82)
	  at oracle.net.ns.NIOPacket.readFromSocketChannel(NIOPacket.java:139)
	  at oracle.net.ns.NIOPacket.readFromSocketChannel(NIOPacket.java:101)
	  at oracle.net.ns.NIONSDataChannel.readDataFromSocketChannel(NIONSDataChannel.java:80)
	  at oracle.jdbc.driver.T4CMAREngineNIO.prepareForReading(T4CMAREngineNIO.java:98)
	  at oracle.jdbc.driver.T4CMAREngineNIO.unmarshalUB1(T4CMAREngineNIO.java:534)
	  at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:485)
	  at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:252)
	  at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:612)
	  at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:213)
	  at oracle.jdbc.driver.T4CStatement.fetch(T4CStatement.java:1009)
	  at oracle.jdbc.driver.OracleStatement.fetchMoreRows(OracleStatement.java:3353)
	  at oracle.jdbc.driver.InsensitiveScrollableResultSet.fetchMoreRows(InsensitiveScrollableResultSet.java:736)
	  at oracle.jdbc.driver.InsensitiveScrollableResultSet.absoluteInternal(InsensitiveScrollableResultSet.java:692)
	  at oracle.jdbc.driver.InsensitiveScrollableResultSet.next(InsensitiveScrollableResultSet.java:406)
	  - locked <0xdcb> (a oracle.jdbc.driver.T4CConnection)
	  at io.debezium.jdbc.JdbcConnection.readSchema(JdbcConnection.java:1003)
	  at io.debezium.connector.oracle.OracleConnection.readSchema(OracleConnection.java:101)

      The following code:

      io.debezium.jdbc.JdbcConnection.java
      try (ResultSet columnMetadata = metadata.getColumns(databaseCatalog, schemaNamePattern, null, null)) {
    while (columnMetadata.next()) {
        String catalogName = columnMetadata.getString(1);
        String schemaName = columnMetadata.getString(2);
        String tableName = columnMetadata.getString(3);
        TableId tableId = new TableId(catalogName, schemaName, tableName);

        ...
    }
}

      reads all columns from given database schema, it is very slow in my case.

      I suggest to read columns only for captured tables.

              Unassigned Unassigned
              andrey.pustovetov@gmail.com Andrey Pustovetov (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: