[2022-02-28 10:33:30,667] INFO Snapshot step 7 - Skipping snapshotting of data (io.debezium.relational.RelationalSnapshotChangeEventSource:132) [2022-02-28 10:33:30,669] INFO Snapshot - Final stage (io.debezium.pipeline.source.AbstractSnapshotChangeEventSource:88) [2022-02-28 10:33:30,669] INFO Snapshot ended with SnapshotResult [status=COMPLETED, offset=OracleOffsetContext [scn=989403998940]] (io.debezium.pipeline.ChangeEventSourceCoordinator:155) [2022-02-28 10:33:30,671] INFO Connected metrics set to 'true' (io.debezium.pipeline.ChangeEventSourceCoordinator:233) [2022-02-28 10:33:30,671] INFO Starting streaming (io.debezium.pipeline.ChangeEventSourceCoordinator:172) [2022-02-28 10:33:31,327] INFO Transaction 4c0005004e800c00 was pending across snapshot boundary. Start SCN = 988623360993, snapshot SCN = 989403998940 (io.debezium.connector.oracle.logminer.LogMinerStreamingChangeEventSource:232) [2022-02-28 10:33:31,327] INFO Transaction af001d0042230000 was pending across snapshot boundary. Start SCN = 988626920258, snapshot SCN = 989403998940 (io.debezium.connector.oracle.logminer.LogMinerStreamingChangeEventSource:232) [2022-02-28 10:33:31,328] INFO Transaction 19000900a5033400 was pending across snapshot boundary. Start SCN = 988577507602, snapshot SCN = 989403998940 (io.debezium.connector.oracle.logminer.LogMinerStreamingChangeEventSource:232) [2022-02-28 10:33:31,328] WARN Transaction 19000900a5033400 was still ongoing while snapshot was taken, but is no longer completely recorded in the archive logs. Events will be lost. Oldest SCN in logs = 988593145268, TX start SCN = 988577507602 (io.debezium.connector.oracle.logminer.LogMinerStreamingChangeEventSource:234) [2022-02-28 10:33:31,328] INFO Transaction a9000600a0230000 was pending across snapshot boundary. Start SCN = 989403623512, snapshot SCN = 989403998940 (io.debezium.connector.oracle.logminer.LogMinerStreamingChangeEventSource:232) [2022-02-28 10:33:31,328] INFO Setting commit SCN to 989403998939 (snapshot SCN - 1) to ensure we don't double-emit events from pre-snapshot transactions. (io.debezium.connector.oracle.logminer.LogMinerStreamingChangeEventSource:249) [2022-02-28 10:33:31,328] INFO Resetting start SCN from 989403998940 (snapshot SCN) to 988593145268 (start of oldest complete pending transaction) (io.debezium.connector.oracle.logminer.LogMinerStreamingChangeEventSource:256) [2022-02-28 10:33:51,408] INFO WorkerSourceTask{id=GL_prod-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:424) [2022-02-28 10:33:51,408] INFO WorkerSourceTask{id=GL_prod-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:441) [2022-02-28 10:33:51,411] INFO WorkerSourceTask{id=GL_prod-0} Finished commitOffsets successfully in 3 ms (org.apache.kafka.connect.runtime.WorkerSourceTask:523) [2022-02-28 10:34:51,411] INFO WorkerSourceTask{id=GL_prod-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:424) [2022-02-28 10:34:51,412] INFO WorkerSourceTask{id=GL_prod-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:441) [2022-02-28 10:35:51,413] INFO WorkerSourceTask{id=GL_prod-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:424) [2022-02-28 10:35:51,413] INFO WorkerSourceTask{id=GL_prod-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:441) [2022-02-28 10:36:51,414] INFO WorkerSourceTask{id=GL_prod-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:424) [2022-02-28 10:36:51,415] INFO WorkerSourceTask{id=GL_prod-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:441) [2022-02-28 10:37:51,415] INFO WorkerSourceTask{id=GL_prod-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:424) [2022-02-28 10:37:51,415] INFO WorkerSourceTask{id=GL_prod-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:441) [2022-02-28 10:38:51,416] INFO WorkerSourceTask{id=GL_prod-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:424) [2022-02-28 10:38:51,416] INFO WorkerSourceTask{id=GL_prod-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:441) [2022-02-28 10:39:51,417] INFO WorkerSourceTask{id=GL_prod-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:424) [2022-02-28 10:39:51,417] INFO WorkerSourceTask{id=GL_prod-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:441) [2022-02-28 10:40:51,417] INFO WorkerSourceTask{id=GL_prod-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:424) [2022-02-28 10:40:51,418] INFO WorkerSourceTask{id=GL_prod-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:441) [2022-02-28 10:41:51,418] INFO WorkerSourceTask{id=GL_prod-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:424) [2022-02-28 10:41:51,418] INFO WorkerSourceTask{id=GL_prod-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:441) [2022-02-28 10:42:51,419] INFO WorkerSourceTask{id=GL_prod-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:424) [2022-02-28 10:42:51,419] INFO WorkerSourceTask{id=GL_prod-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:441) [2022-02-28 10:43:51,420] INFO WorkerSourceTask{id=GL_prod-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:424) [2022-02-28 10:43:51,420] INFO WorkerSourceTask{id=GL_prod-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:441) [2022-02-28 10:44:51,420] INFO WorkerSourceTask{id=GL_prod-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:424) [2022-02-28 10:44:51,421] INFO WorkerSourceTask{id=GL_prod-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:441) [2022-02-28 10:45:51,421] INFO WorkerSourceTask{id=GL_prod-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:424) [2022-02-28 10:45:51,421] INFO WorkerSourceTask{id=GL_prod-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:441) [2022-02-28 10:46:51,423] INFO WorkerSourceTask{id=GL_prod-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:424) [2022-02-28 10:46:51,424] INFO WorkerSourceTask{id=GL_prod-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:441) [2022-02-28 10:47:51,424] INFO WorkerSourceTask{id=GL_prod-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:424) [2022-02-28 10:47:51,425] INFO WorkerSourceTask{id=GL_prod-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:441) [2022-02-28 10:48:51,425] INFO WorkerSourceTask{id=GL_prod-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:424) [2022-02-28 10:48:51,426] INFO WorkerSourceTask{id=GL_prod-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:441) [2022-02-28 10:49:51,426] INFO WorkerSourceTask{id=GL_prod-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:424) [2022-02-28 10:49:51,427] INFO WorkerSourceTask{id=GL_prod-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:441) [2022-02-28 10:50:51,428] INFO WorkerSourceTask{id=GL_prod-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:424) [2022-02-28 10:50:51,428] INFO WorkerSourceTask{id=GL_prod-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:441) [2022-02-28 10:51:19,679] INFO 1 records sent during previous 00:18:13.221, last recorded offset: {commit_scn=989403998939, transaction_id=null, snapshot_pending_tx=4c0005004e800c00:988623360993,af001d0042230000:988626920258,19000900a5033400:988577507602,a9000600a0230000:989403623512, snapshot_scn=989403998940, scn=988593165267} (io.debezium.connector.common.BaseSourceTask:182) [2022-02-28 10:51:29,398] ERROR Mining session stopped due to the {} (io.debezium.connector.oracle.logminer.LogMinerHelper:113) java.lang.IllegalStateException: None of log files contains offset SCN: 988593165267, re-snapshot is required. at io.debezium.connector.oracle.logminer.LogMinerHelper.setLogFilesForMining(LogMinerHelper.java:96) at io.debezium.connector.oracle.logminer.LogMinerStreamingChangeEventSource.initializeRedoLogsForMining(LogMinerStreamingChangeEventSource.java:320) at io.debezium.connector.oracle.logminer.LogMinerStreamingChangeEventSource.execute(LogMinerStreamingChangeEventSource.java:175) at io.debezium.connector.oracle.logminer.LogMinerStreamingChangeEventSource.execute(LogMinerStreamingChangeEventSource.java:57) at io.debezium.pipeline.ChangeEventSourceCoordinator.streamEvents(ChangeEventSourceCoordinator.java:173) at io.debezium.pipeline.ChangeEventSourceCoordinator.executeChangeEventSources(ChangeEventSourceCoordinator.java:140) at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:109) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) [2022-02-28 10:51:29,400] ERROR Producer failure (io.debezium.pipeline.ErrorHandler:35) java.lang.IllegalStateException: None of log files contains offset SCN: 988593165267, re-snapshot is required. at io.debezium.connector.oracle.logminer.LogMinerHelper.setLogFilesForMining(LogMinerHelper.java:96) at io.debezium.connector.oracle.logminer.LogMinerStreamingChangeEventSource.initializeRedoLogsForMining(LogMinerStreamingChangeEventSource.java:320) at io.debezium.connector.oracle.logminer.LogMinerStreamingChangeEventSource.execute(LogMinerStreamingChangeEventSource.java:175) at io.debezium.connector.oracle.logminer.LogMinerStreamingChangeEventSource.execute(LogMinerStreamingChangeEventSource.java:57) at io.debezium.pipeline.ChangeEventSourceCoordinator.streamEvents(ChangeEventSourceCoordinator.java:173) at io.debezium.pipeline.ChangeEventSourceCoordinator.executeChangeEventSources(ChangeEventSourceCoordinator.java:140) at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:109) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) [2022-02-28 10:51:29,400] INFO startScn=988593165267, endScn=988593186267 (io.debezium.connector.oracle.logminer.LogMinerStreamingChangeEventSource:200) [2022-02-28 10:51:29,401] INFO Streaming metrics dump: OracleStreamingChangeEventSourceMetrics{currentScn=989404259545, oldestScn=null, committedScn=null, offsetScn=null, logMinerQueryCount=1, totalProcessedRows=12514962, totalCapturedDmlCount=0, totalDurationOfFetchingQuery=PT1.415S, lastCapturedDmlCount=0, lastDurationOfFetchingQuery=PT1.415S, maxCapturedDmlCount=0, maxDurationOfFetchingQuery=PT1.415S, totalBatchProcessingDuration=PT17M8.579S, lastBatchProcessingDuration=PT17M8.579S, maxBatchProcessingThroughput=0, currentLogFileName=[+DATA/GL/ONLINELOG/group_11.324.1064674631, +FRA/GL/ONLINELOG/group_11.2416.1064674645], minLogFilesMined=2, maxLogFilesMined=2, redoLogStatus=[+DATA/GL/ONLINELOG/group_14.291.1064674739 | ACTIVE, +FRA/GL/ONLINELOG/group_14.2640.1064674753 | ACTIVE, +DATA/GL/ONLINELOG/group_8.321.1064674513 | ACTIVE, +FRA/GL/ONLINELOG/group_8.2419.1064674525 | ACTIVE, +DATA/GL/ONLINELOG/group_9.322.1064674537 | ACTIVE, +FRA/GL/ONLINELOG/group_9.2418.1064674549 | ACTIVE, +FRA/GL/ONLINELOG/group_13.2414.1064674693 | ACTIVE, +FRA/GL/ONLINELOG/group_10.2417.1064674619 | ACTIVE, +DATA/GL/ONLINELOG/group_10.323.1064674607 | ACTIVE, +DATA/GL/ONLINELOG/group_13.326.1064674681 | ACTIVE, +FRA/GL/ONLINELOG/group_11.2416.1064674645 | CURRENT, +DATA/GL/ONLINELOG/group_11.324.1064674631 | CURRENT, +DATA/GL/ONLINELOG/group_12.325.1064674657 | INACTIVE, +FRA/GL/ONLINELOG/group_12.2415.1064674669 | INACTIVE], switchCounter=43, batchSize=22000, millisecondToSleepBetweenMiningQuery=600, hoursToKeepTransaction=1, networkConnectionProblemsCounter0, batchSizeDefault=20000, batchSizeMin=1000, batchSizeMax=100000, sleepTimeDefault=1000, sleepTimeMin=0, sleepTimeMax=3000, sleepTimeIncrement=200, totalParseTime=PT0S, totalStartLogMiningSessionDuration=PT40.683S, lastStartLogMiningSessionDuration=PT40.683S, maxStartLogMiningSessionDuration=PT40.683S, totalProcessTime=PT17M8.579S, minBatchProcessTime=PT17M8.579S, maxBatchProcessTime=PT17M8.579S, totalResultSetNextTime=PT15M7.625S, lagFromTheSource=DurationPT0S, maxLagFromTheSourceDuration=PT0S, minLagFromTheSourceDuration=PT0S, lastCommitDuration=PT0S, maxCommitDuration=PT0S, activeTransactions=0, rolledBackTransactions=0, committedTransactions=0, abandonedTransactionIds=[], rolledbackTransactionIds=[], registeredDmlCount=0, committedDmlCount=0, errorCount=1, warningCount=0, scnFreezeCount=0, unparsableDdlCount=0, miningSessionUserGlobalAreaMemory=59993512, miningSessionUserGlobalAreaMaxMemory=96172192, miningSessionProcessGlobalAreaMemory=116793896, miningSessionProcessGlobalAreaMaxMemory=116793896} (io.debezium.connector.oracle.logminer.LogMinerStreamingChangeEventSource:201) [2022-02-28 10:51:29,401] INFO Offsets: OracleOffsetContext [scn=988593165267] (io.debezium.connector.oracle.logminer.LogMinerStreamingChangeEventSource:202) [2022-02-28 10:51:29,401] INFO Finished streaming (io.debezium.pipeline.ChangeEventSourceCoordinator:174) [2022-02-28 10:51:29,401] INFO Connected metrics set to 'false' (io.debezium.pipeline.ChangeEventSourceCoordinator:233) [2022-02-28 10:51:29,702] INFO WorkerSourceTask{id=GL_prod-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:424) [2022-02-28 10:51:29,703] INFO WorkerSourceTask{id=GL_prod-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:441) [2022-02-28 10:51:29,707] INFO WorkerSourceTask{id=GL_prod-0} Finished commitOffsets successfully in 4 ms (org.apache.kafka.connect.runtime.WorkerSourceTask:523) [2022-02-28 10:51:29,707] ERROR WorkerSourceTask{id=GL_prod-0} Task threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask:186) org.apache.kafka.connect.errors.ConnectException: An exception occurred in the change event producer. This connector will be stopped. at io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:50) at io.debezium.connector.oracle.logminer.LogMinerStreamingChangeEventSource.execute(LogMinerStreamingChangeEventSource.java:197) at io.debezium.connector.oracle.logminer.LogMinerStreamingChangeEventSource.execute(LogMinerStreamingChangeEventSource.java:57) at io.debezium.pipeline.ChangeEventSourceCoordinator.streamEvents(ChangeEventSourceCoordinator.java:173) at io.debezium.pipeline.ChangeEventSourceCoordinator.executeChangeEventSources(ChangeEventSourceCoordinator.java:140) at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:109) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.IllegalStateException: None of log files contains offset SCN: 988593165267, re-snapshot is required. at io.debezium.connector.oracle.logminer.LogMinerHelper.setLogFilesForMining(LogMinerHelper.java:96) at io.debezium.connector.oracle.logminer.LogMinerStreamingChangeEventSource.initializeRedoLogsForMining(LogMinerStreamingChangeEventSource.java:320) at io.debezium.connector.oracle.logminer.LogMinerStreamingChangeEventSource.execute(LogMinerStreamingChangeEventSource.java:175) ... 9 more [2022-02-28 10:51:29,708] ERROR WorkerSourceTask{id=GL_prod-0} Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask:187) [2022-02-28 10:51:29,708] INFO Stopping down connector (io.debezium.connector.common.BaseSourceTask:238)