-
Bug
-
Resolution: Done
-
Blocker
-
None
-
None
-
False
-
None
-
False
-
Hello,
Debezium after a reboot, during the process of looking for a starting position in WAL never found it.
It kept throwing up logs for 10 hours:
'INFO Postgres|postgres-connector-task Streaming requested from LSN LSN{2AEAF/E2245128}, received LSN LSN{2AF65/B84E7808} identified as already processed [io.debezium.connector.postgresql.connection.AbstractMessageDecoder]'
while simultaneously committing progress, which ends up cleaning up WAL. Unfortunately, we lost a lot of data because the searching process did not break with any exception.
It looks as if the condition in WalPositionLocator has never been met:
if (startStreamingLsn == null || startStreamingLsn.equals(lsn))
We also observed that the offsets commited to kafka at restart time were not increasing. E.g. 98 - 100 - 99 - 101
Unfortunately the logs were lost after restarting the container (restart also solved the problem)
We are using debezium 1.5 with on-premise postgres (with huge load and hundreds of tables, complicated procedures and so on) 14 with logical replication (pgout)
Do you have any idea what happened and how to prevent it in the future?