-
Bug
-
Resolution: Done
-
Critical
-
0.9.0.CR1
-
None
These are (commitLsn, changeLsn) pairs from Kafka:
0x000443D80073D0DA0005 0x000443D80073D037005E 0x000443D80073D0DA0005 0x000443D80073D0370060 0x000443D80073D0DA0005 0x000443D80073D0370062 0x000443D80073D0DA0005 0x000443D80073D0370064 0x000443D80073D0DA0005 0x000443D80073D0370066 0x000443D80073D0DA0005 0x000443D80073D0370068 0x000443D80073D0DA0005 0x000443D80073D037006A 0x000443D80073D0DA0005 0x000443D80073D037006C 0x000443D80073D0DA0005 0x000443D80073D037006E 0x000443D80073D0DA0005 0x000443D80073D037006F 0x000443D80073D0DA0005 0x000443D80073D0370075 0x000443D80073D0DA0005 0x000443D80073D0370077 0x000443D80073D0DA0005 0x000443D80073D0370079 0x000443D80073D0DA0005 0x000443D80073D037007A 0x000443D80073D0DA0005 0x000443D80073D0370080 0x000443D80073D0DA0005 0x000443D80073D0370082 0x000443D80073D0DA0005 0x000443D80073D0370084 0x000443D80073D0DA0005 0x000443D80073D0370086 0x000443D80073D0DA0005 0x000443D80073D0370088 0x000443D80073D0DA0005 0x000443D80073D037008A 0x000443D80073D0DA0005 0x000443D80073D037008C 0x000443D80073D0DA0005 0x000443D80073D037008E 0x000443D80073D0DA0005 0x000443D80073D0370090 0x000443D80073D0DA0005 0x000443D80073D0370092 0x000443D80073D0DA0005 0x000443D80073D0370094 0x000443D80073D0DA0005 0x000443D80073D0370096 0x000443D80073D0DA0005 0x000443D80073D0370098 0x000443D80073D0DA0005 0x000443D80073D037009A 0x000443D80073D0DA0005 0x000443D80073D037009C 0x000443D80073D0DA0005 0x000443D80073D037009E 0x000443D80073D0DA0005 0x000443D80073D03700A0 0x000443D80073D0DA0005 0x000443D80073D03700A2 0x000443D80073D0DA0005 0x000443D80073D03700A4 0x000443D80073D0DA0005 0x000443D80073D03700A6 0x000443D80073D0DA0005 0x000443D80073D03700A8 0x000443D80073D1F30001 0x000443D80073D0EC0016 0x000443D80073D1F30001 0x000443D80073D0EC001C 0x000443D80073D1F30001 0x000443D80073D0EC001E
These are (commitLsn, changeLsn) pairs from capture instance:
0x000443D80073D0DA0005 0x000443D80073D037005E 0x000443D80073D0DA0005 0x000443D80073D0370060 0x000443D80073D0DA0005 0x000443D80073D0370062 0x000443D80073D0DA0005 0x000443D80073D0370064 0x000443D80073D0DA0005 0x000443D80073D0370066 0x000443D80073D0DA0005 0x000443D80073D0370068 0x000443D80073D0DA0005 0x000443D80073D037006A 0x000443D80073D0DA0005 0x000443D80073D037006C 0x000443D80073D0DA0005 0x000443D80073D037006E 0x000443D80073D0DA0005 0x000443D80073D037006F 0x000443D80073D0DA0005 0x000443D80073D0370075 0x000443D80073D0DA0005 0x000443D80073D0370077 0x000443D80073D0DA0005 0x000443D80073D0370079 0x000443D80073D0DA0005 0x000443D80073D037007A 0x000443D80073D0DA0005 0x000443D80073D0370080 0x000443D80073D0DA0005 0x000443D80073D0370082 0x000443D80073D0DA0005 0x000443D80073D0370084 0x000443D80073D0DA0005 0x000443D80073D0370086 0x000443D80073D0DA0005 0x000443D80073D0370088 0x000443D80073D0DA0005 0x000443D80073D037008A 0x000443D80073D0DA0005 0x000443D80073D037008C 0x000443D80073D0DA0005 0x000443D80073D037008E 0x000443D80073D0DA0005 0x000443D80073D0370090 0x000443D80073D0DA0005 0x000443D80073D0370092 0x000443D80073D0DA0005 0x000443D80073D0370094 0x000443D80073D0DA0005 0x000443D80073D0370096 0x000443D80073D0DA0005 0x000443D80073D0370098 0x000443D80073D0DA0005 0x000443D80073D037009A 0x000443D80073D0DA0005 0x000443D80073D037009C 0x000443D80073D0DA0005 0x000443D80073D037009E 0x000443D80073D0DA0005 0x000443D80073D03700A0 0x000443D80073D0DA0005 0x000443D80073D03700A2 0x000443D80073D0DA0005 0x000443D80073D03700A4 0x000443D80073D0DA0005 0x000443D80073D03700A6 0x000443D80073D0DA0005 0x000443D80073D03700A8 <--- The last record in Kafka with commitLsn = 0x000443D80073D0DA0005 0x000443D80073D0DA0005 0x000443D80073D03700AA 0x000443D80073D0DA0005 0x000443D80073D03700AC 0x000443D80073D0DA0005 0x000443D80073D03700AE 0x000443D80073D0DA0005 0x000443D80073D03700B0 0x000443D80073D0DA0005 0x000443D80073D03700B2 0x000443D80073D0DA0005 0x000443D80073D03700B4 0x000443D80073D0DA0005 0x000443D80073D03700B5 0x000443D80073D0DA0005 0x000443D80073D03700BB 0x000443D80073D0DA0005 0x000443D80073D03700BD 0x000443D80073D0DA0005 0x000443D80073D03700BF 0x000443D80073D0DA0005 0x000443D80073D03700C1 0x000443D80073D0DA0005 0x000443D80073D03700C3 0x000443D80073D0DA0005 0x000443D80073D03700C5 0x000443D80073D0DA0005 0x000443D80073D03700C7 0x000443D80073D0DA0005 0x000443D80073D03700C9 0x000443D80073D0DA0005 0x000443D80073D03700CB 0x000443D80073D0DA0005 0x000443D80073D03700CD 0x000443D80073D0DA0005 0x000443D80073D03700CF 0x000443D80073D0DA0005 0x000443D80073D03700D0 0x000443D80073D0DA0005 0x000443D80073D03700D6 0x000443D80073D0DA0005 0x000443D80073D03700D8 0x000443D80073D0DA0005 0x000443D80073D03700D9 0x000443D80073D1F30001 0x000443D80073D0EC0016 0x000443D80073D1F30001 0x000443D80073D0EC001C 0x000443D80073D1F30001 0x000443D80073D0EC001E
This is the last offset saved before KConnect restart (note it is exactly the same as commitLsn of the last record saved in Kafka):
{"commit_lsn":"000443d8:0073d0da:0005","change_lsn":"000443d8:0073d037:00a8"}
2019-01-31 11:48:57,445 INFO Found previous offset SqlServerOffsetContext [sourceInfoSchema=Schema{io.debezium.connector.sqlserver.Source:STRUCT}, sourceInfo=SourceInfo [serverName=XXX, changeLsn=000443d8:0073d037:00a8, commitLsn=000443d8:0073d0da:0005, snapshot=false, sourceTime=null], partition={server=XXX}, snapshotCompleted=false] (io.debezium.connector.sqlserver.SqlServerConnectorTask:142)
2019-01-31 11:49:00,613 INFO Last position recorded in offsets is 000443d8:0073d0da:0005(000443d8:0073d037:00a8) (io.debezium.connector.sqlserver.SqlServerStreamingChangeEventSource:72)
At first glance it seems, that the remaining, unprocessed part with commitLsn=0x000443D80073D0DA0005 was skipped because in the very first iteration commitLsn is increased by one:
// Reading interval is inclusive so we need to move LSN forward final Lsn fromLsn = lastProcessedPosition.getCommitLsn().isAvailable() ? connection.incrementLsn(lastProcessedPosition.getCommitLsn()) : lastProcessedPosition.getCommitLsn();
