Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-2855

Missing log file error when current SCN differs from snapshotted in Oracle connector and Logminer

    XMLWordPrintable

    Details

      Description

      Hi there.

      We are testing Debezium internally with some existing Oracle 19 database. We are using Logminer and the Debezium Oracle Connector.

      Somehow, last week we got into a state where we would always hit a "missing log file" error immediately after installing the connector. The error message was very similar to the reported in DBZ-2759 but the connector wouldn't really seem to even have the chance to start. Here is the stack trace for reference:

      [2020-12-14 21:06:19,488] ERROR Mining session stopped due to the {} (io.debezium.connector.oracle.logminer.LogMinerHelper:515)
      java.sql.SQLException: ORA-01291: missing log file
      ORA-06512: at "SYS.DBMS_LOGMNR", line 72
      ORA-06512: at line 1	at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:509)
      	at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:461)
      	at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:1104)
      	at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:553)
      	at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:269)
      	at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:655)
      	at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:265)
      	at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:86)
      	at oracle.jdbc.driver.T4CCallableStatement.executeForRows(T4CCallableStatement.java:965)
      	at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1205)
      	at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3666)
      	at oracle.jdbc.driver.T4CCallableStatement.executeInternal(T4CCallableStatement.java:1358)
      	at oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:3778)
      	at oracle.jdbc.driver.OracleCallableStatement.execute(OracleCallableStatement.java:4251)
      	at oracle.jdbc.driver.OraclePreparedStatementWrapper.execute(OraclePreparedStatementWrapper.java:1081)
      	at io.debezium.connector.oracle.logminer.LogMinerHelper.executeCallableStatement(LogMinerHelper.java:571)
      	at io.debezium.connector.oracle.logminer.LogMinerHelper.startLogMining(LogMinerHelper.java:251)
      	at io.debezium.connector.oracle.logminer.LogMinerStreamingChangeEventSource.execute(LogMinerStreamingChangeEventSource.java:180) 

      After forking the connector, digging in the source and adding some extra tracing logs, we finally found what was going on. Essentially it looks like at startup a list of logging files is grabbed from the last SCN offset, but that list does not necessarily match the current log files if you DB team has been doing some changes. As the current log files doesn't change either, then the Debezium will not detect a log file switch  and will not trigger a reload. Hence the current log files are never added to the LogMiner session and the next time you get to a SCN living within the current log file you end up with the "missing log file" exception.

      We verified that by adding the following code in LogMinerHelper seems to fix this issue as it also considers current log files when reloading, but I have the feeling that this might not cover all the possible edge cases there, so rather than sending a PR I've decided to just open this bug and get some feedback here. Also the null metrics parameter kind of sucks there, but was really meant to quickly prove the solution.

      List<String> currentFileNames = getCurrentRedoLogFiles(connection, null).stream()
         .filter(e -> !logFilesNames.contains(e)).collect(Collectors.toList());
      if (currentFileNames.size() > 0){     
          logFilesNames.addAll(currentFileNames); 
      }
       

      Cheers,

      Martin

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              ccranfor Chris Cranford
              Reporter:
              martper2 Martin Perez
              Votes:
              1 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: