Using online mode, not RAC.
- Snapshot completed and streaming started.
- For a few days everything was streaming/syncing along well.
- Then after 3 days my independent rowcount rec process created an alert to show that my oracle source db had 5 more rows than what kafka received for 1 specific table. But debezium never logged such issue!
- After digging into what the scn on oracle side was for couple of the missing rows and then looking at v$logmnr_contents for that scn i saw some rows like this :
|UNSUPPORTED||255||0||ABC||REDACT||REDACT||2||TABLE||REDACT||AALGiNAArAAD/T5AAJ||UNKNOWN||UNKNOWN||UNKNOWN||0||0||0||UNKNOWN||1||3||8005||188433||488||33||2522166||40||29101||51||43||1045753||289128||0||2910349||Unsupported||Unsupported||0x001f45.0002e011.01e8||0||0||Object or Data type Unsupported||2||823020||823021||0||5.24722E+11||UNKNOWN||0||1|
|UNSUPPORTED||255||0||ABC||REDACT||REDACT||2||TABLE||REDACT||AALGiNAAAAAAAAAAAA||UNKNOWN||UNKNOWN||UNKNOWN||0||0||0||UNKNOWN||1||4||8005||188436||56||33||2522166||40||29101||33||43||1045753||289128||0||2910349||Unsupported||Unsupported||0x001f45.0002e014.0038||0||0||Object or Data type Unsupported||2||823022||823023||0||5.24722E+11||UNKNOWN||0||1|
Some actions I can think of:
- Make debezium throw an exception whenever it encounters an 'unsupported' row during mining, currently there was not even log to say it encountered this
- Make debezium docs refer to https://docs.oracle.com/en/database/oracle/oracle-database/19/sutil/oracle-logminer-utility.html#GUID-8A4F98EC-C233-4471-BFF9-9FB35EF5AD0D (23.13.2
Unsupported Data Types and Table Storage Attributes) and part about tables/cols > 30 characters long
- But the curious thing is I don't think I am using any of these types of things like BFILE..etc
- And to make things even stranger, looking at a secondtable, it has some rows as supported and some unsupported!? so the restrictions are not just table/column definition level but also row dml level? which makes me think there is oracle side bug or some more hidden restrictions oracle is not telling us...
|secondtable||2||Object or Data type Unsupported||1|
a) make debezium automatically detect unsupported tables before snapshot starts, at least according to the criteria in the docs...which may just be partial criteria