Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-8984

FileOffsetBackingStore can cause corrupted offsets when process terminated mid-flush

XMLWordPrintable

    • False
    • Hide

      None

      Show
      None
    • False

      See https://debezium.zulipchat.com/#narrow/channel/302529-community-general/topic/Overcoming.20Debezium.20Engine.20Offsets.2Edat.20file.20corruption/near/514916966

      In short, when the FileOffsetBackingStore#save method is called, the ObjectOutputStream may be mid-way through the write call when the Java process is terminated, and this can lead to corrupt offset details.

      We should consider reworking the FileOffsetBackingStore call so that we write the contents of the offsets to a secondary file and perform an atomic_move at the end of the operation to guarantee that if the Java task is stopped before the file is completely written, that on restart the offset details don't become corrupt and unusable.

      As an added feature suggested by Jiri, we could consider supporting a rolling approach where we maintain several prior instances of the offsets, see the Zulip thread.

              Unassigned Unassigned
              ccranfor@redhat.com Chris Cranford
              Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: