Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-5597

Message with LSN foo larger than expected LSN bar

XMLWordPrintable

    • False
    • None
    • False
    • Hide

      So far we have not been able to reproduce this reliably. The error was seen twice but the trigger is completely unknown

      Show
      So far we have not been able to reproduce this reliably. The error was seen twice but the trigger is completely unknown

      Setup Details

      Debezium Version: 1.9.5.FINAL

      Debezium Runtime: Debezium/Connect Docker container in standalone mode; Deployed on-prem on bare metal, talking to PG on the same machine. Kafka cluster in a managed cloud.

      Postgres version: 12:12

      Postgres Logical Decoder: pgoutput

      Description

      I encountered this Debezium exception that I need some help deciphering -
       

      Message with LSN 'LSN{22E3/5E30CDD8}' larger than expected LSN 'LSN{22E3/5E30CAB8}'

      The happens right after Debezium connector restarts after a connector failure, where the connector failed to talk to PG. I am attaching the log file that has all the log messages from the connector restart (after a failure), leading up to this exception.

      According to the logs,

      • the PG confirmed_flush_lsn was 22E3/29F39960 (line 2758 in log)
      • the committed LSN in Kafka was 22E3/5E3118F8 (line 2759)
      • the last processed LSN in Kafka was 22E3/5E30C0E0 (line 2759)

      The error suggests that PG sends LSN 22E3/5E30CDD8 right after connecting (line 2935) and then (presumably) sends 22E3/5E30CAB8 right after that (line 2936). It establishes 22E3/5E30CAB8 as the restart point for streaming because it figures 22E3/5E30CAB8 is the first greater offset after 22E3/5E30C0E0 (the last processed offset according to the Kafka internal topic). However, it then complains that 22E3/5E30CAB is less than 22E3/5E30CDD8. I would appreciate help in understanding why PG would send a second message with an LSN that is lesser than the one it previously sent.

      Thanks!

       

      Note: This was filed on advice from the thread started here - https://debezium.zulipchat.com/#narrow/stream/302529-users/topic/Message.20with.20LSN.20foo.20larger.20than.20expected.20LSN.20bar

        1. current-358-09-10.log
          2.66 MB
        2. current-385-09-14.log
          2.42 MB
        3. current-413-09-13.log
          1.23 MB
        4. current-415-09-14.log
          1.03 MB
        5. current-415-09-14-verified.tgz
          177 kB
        6. current-59-09-10.log
          3.69 MB
        7. dbz.patch
          2 kB
        8. debezium-connector-postgres-1.9.0-SNAPSHOT-plugin.tar-1.gz
          6.20 MB
        9. debezium-lsn-fail1.log
          837 kB

            Unassigned Unassigned
            gamathur Gaurav Mathur (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated:
              Resolved: