Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-5291

Connector ignores database history if binlog file number exceeds 999999

    XMLWordPrintable

Details

    Description

      In order to make your issue reports as actionable as possible, please provide the following information, depending on the issue type.

      Bug report

      For bug reports, provide this information, please:

      What is the connector configuration?

      Irrelevant

      What is the captured database version and mode of depoyment?

      MySQL compatible Aurora (announces itself as MySQL 5.7.35).

      What behaviour do you expect?

      Messages in the history topic are not ignored.

      What behaviour do you see?

      Messages in the history topic are ignored.

      Do you see the same behaviour using the latest relesead Debezium version?

      I haven't tested but judging by the code this problem exists in all versions.

      Do you have the connector logs, ideally from start till finish?

      No

      How to reproduce the issue using our tutorial deployment?

      If the binlog filename number exceeds 999999, it will consist of 7 numbers, e.g. 1000000 (source). When reading database history, MySqlHistoryRecordComparator compares the file numbers in the messages with the ones in the stored offsets lexicographically. This way, the string "999999" (in the message from database history) is greater than the string "1000000" (in the committed offset), so all DDLs captured from the binlog files with the numbers between 100001 and 999999 will be ignored resulting in an incomplete in-memory schema.

      Implementation ideas

      Replace lexicographic comparison of binlog filenames with numeric.

      Attachments

        Activity

          People

            Unassigned Unassigned
            sergeimorozov Sergei Morozov
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: