Connector logs 'Disabling skipping changes due to not monotonic order of changes' frequently. What are the implications of this message?

Hi there! 

This is a question on the occurrence of a log.

In Sql Server Debezium Connector v1.4.0, we see a high frequency of this log: 'Disabling skipping changes due to not monotonic order of changes'. The tables the connector is reading from has a high volume of transactions. I want to understand the implications of this message.

Concretely:

• The logs show that Disabling skipping changes due to not monotonic order of changes happening every couple of minutes in the main execution loop of the CDC connector. This log indicates that changesStoppedBeingMonotonic.set(true); https://github.com/debezium/debezium/blob/master/debezium-connector-sqlserver/src/main/java/io/debezium/connector/sqlserver/SqlServerStreamingChangeEventSource.java#L205

• Similarly the log Resetting changesStoppedBeingMonotonic as transaction changes is seen with a similar frequency- occurring in-between Disabling skipping changes due to not monotonic order of changes' . This second log indicates that changesStoppedBeingMonotonic.set(false); https://github.com/debezium/debezium/blob/master/debezium-connector-sqlserver/src/main/java/io/debezium/connector/sqlserver/SqlServerStreamingChangeEventSource.java#L199
• This change was introduced in this PR DBZ-2329 Fix for: Change events lost when connnector is restarted while processing transaction with PK update by korzenek - Pull Request #1691 - debezium/debezium and this issue https://issues.redhat.com/browse/DBZ-2329
• According to this issue, the changes were introduced because sometimes during update of multiple primary keys order of change_lsn is not monotonic.
  • However in the PR it is fleshed out some more, and suggests that this situation can happen for all transactions which were aborted in the middle with offset commit.
  • Per the pr the problem applies only to a single TX as no more than one TX can be aborted in the middle. i.e the log Disabling skipping changes due to not monotonic order of changes a transaction has been aborted in the middle with an offset commit - and so the connector is going to log it but process the change capture so as to not lose any data, and once the log Resetting changesStoppedBeingMonotonic as transaction changes is logged, we are given the signal that a new transaction has started.
  • IIUC, the fact that we're seeing continuous logs alternating between these two messages, must mean that the connector must be seeing a lot of transactions, that do not end before the offset is committed.
    • Since the logs are suggesting that we're not skipping over these changes, I think it might be okay that it is happening.

Concerns.

• The one thing I don't understand about DBZ-2329 is that in the PR they suggest that the logs Disabling skipping changes due to not monotonic order of changes and Resetting changesStoppedBeingMonotonic as transaction changes should not happen often and only happen after a restart.
  • See this comment https://github.com/debezium/debezium/pull/1691#discussion_r457906308 (though this comment was made by a reviewer and not an author so may not be valid)
  • Similarly the comments in the source suggest this should happen only after a connector restart. https://github.com/debezium/debezium/blob/master/debezium-connector-sqlserver/src/main/java/io/debezium/connector/sqlserver/SqlServerStreamingChangeEventSource.java#L202
  • The caveat here is that whole idea that this situation can happen for all transactions which were aborted in the middle with offset commit only came up during review and perhaps these comments were not updated. I can't tell.
• These logs are not in sync with the connector restarts. In the logs we only see the connector being stopped a couple times, but the logs show up a lot, a lot more.

Question:

• Are the logs Disabling skipping changes due to not monotonic order of changes and Resetting changesStoppedBeingMonotonic as transaction changes only supposed to happen after a restart? Or are they indicative of a high volume of transactions?

• Are they indicative of something going on in this set up that is undesirable? Or is it okay?