Status: Closed (View Workflow)
- Setup an Oracle DB configured for logminer, with a table that will be captured.
- Start inserting records in the database, one by one, each in its own transaction.
- While inserting records, create a connector.
- Check if all inserted records are captured by Debezium, one should be missing.
(more details given in the issue)Setup an Oracle DB configured for logminer, with a table that will be captured. Start inserting records in the database, one by one, each in its own transaction. While inserting records, create a connector. Check if all inserted records are captured by Debezium, one should be missing. (more details given in the issue)
This bug seems to be related to https://issues.redhat.com/browse/DBZ-4367.
In short: it seems that it is still possible for changes to go missing during the switch from snapshot to streaming mode, though it is much harder to do than before.
Oracle connector 1.9.1.Final
The connector configuration has been added as an attachment. Note that the we are using the embedded engine.
An Oracle 19c database, running locally in a docker container.
All records inserted into the database should be captured.
A record is missing, right when the connector switches from snapshot to streaming mode.
I haven't had the time yet to test with a 2.0.0 alpha release.
The logs have been added as an attachment, the Debezium log level was on TRACE. Since we are using Debezium engine, there will be some logs in there from the host application as well.
There was a lot of junk in these logs, so I pruned them a bit to make it possible to navigate them, for instance I removed the constant:
Let me know if you think anything's missing, I'll add the full logs then.
In this particular case, it is the record with COL1=273 that is missing. 272 was the last record of the snapshot, 274 the first record in streaming mode (see line 726 and onward).
How to reproduce the issue using our tutorial deployment?
In order to reproduce this, we need to insert some records from one thread, while creating the connector from another. Using the examples from the tutorial (after creating a table and adding the supplemental log data):
Have one thread doing something like:
While data is being inserted, create a connector
Important here is to commit every record separately, or at least in very small transactions. The larger the transactions, the less likely this bug is to occur.
In OracleSnapshotChangeEventSource.java, in the determineSnapshotOffset method, we can see the following code:
If a transaction is committed between retrieving the currentScn and the call to getSnapshotPendingTransactions, those changes will not be captured.
I might have a go at creating an integration test for this, as the issue is quite tricky to replicate.