Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-7293

Race condition between TYPE and ROW events in vitess

XMLWordPrintable

    • False
    • None
    • False
    • Moderate

      Bug report

      Sometimes due to race condition in Vitess, ROW and TYPE messages can be mixed up. 

      https://vitess.io/docs/18.0/reference/vreplication/internal/tracker/#caveat
      {{}}

      `Only best-effort versioning can be provided due to races between DDLs and DMLs`

      Exception:

      Caused by: java.lang.IllegalStateException: The number of columns in the ROW event lengths: 11
      lengths: 10
      lengths: -1
      lengths: 33
      lengths: -1
      lengths: 1
      lengths: 7
      lengths: 4
      lengths: 4
      lengths: 19
      lengths: 19
      values: "15967431361389815089403_01b11_V97ZQ2kdYatJ78jovniiQbXm1#817C75120016002023-12-20 09:44:362023-12-20 09:44:36"
       is different from the in-memory table schema columns: {
        id INT64 NOT NULL
        item_id INT64 NOT NULL
        uuid VARBINARY NOT NULL
        orientation INT16 DEFAULT VALUE NULL
        image_no INT32 NOT NULL
        dominant_color CHAR DEFAULT VALUE NULL
        width INT32 DEFAULT VALUE NULL
        height INT32 DEFAULT VALUE NULL
        created_at DATETIME DEFAULT VALUE NULL
        updated_at DATETIME DEFAULT VALUE NULL
      }
      primary key: [id]
      default charset: null 

      What Debezium connector do you use and what version?

      vitess-debezium-connector 2.5

      What is the captured database version and mode of depoyment?

      Vitess 11.0.4

      What behaviour do you expect?

      No exception during schema update

      What behaviour do you see?

      Error

      Do you see the same behaviour using the latest relesead Debezium version?

      Yes

      How to reproduce the issue using our tutorial deployment?

      Quite hard to reproduce, it has to be a race condition from vitess side, usually we get these on huge sharded tables (size in terrabytes).

      Feature request or enhancement

      Implementation ideas

      Maintain backup schema version for a table.

      If we get ROW event with old schema earlier than FIELD event, fallback to schema backup and de-serialize row that way.

      Usually there are only few ROW events which are out-of-order and after getting newest FIELD event fallback is no longer needed.

            Unassigned Unassigned
            simsas@gmail.com Simonas Gelazevicius (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: