Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-5018

Redis Sink - Use Transaction per Stream instead of Pipeline

    XMLWordPrintable

Details

    • Enhancement
    • Resolution: Unresolved
    • Major
    • 2.4-backlog
    • None
    • debezium-server
    • None
    • False
    • None
    • False

    Description

      We used to have the batch implemented via a transaction but changed it to pipeline as in multi sharded deployment of the Redis DB, keys aimed to different shards can't be part of the same transaction.

      With pipelines we can end up with having duplicate records (although we reduced dramatically the chances to get this as part of the DBZ-4966 fix) not to mention we should parse the responses one by one as some of the commands might fail (while transactions can be rollbacked and once it succeeds, all of its commands can be marked as processed).

      We can switch back to transactions by having a transaction per stream (stream name is the key on which the hashtag function is performed tunneled it to the same shard) and guarantee the whole transaction will go to the same shard.

      We will achieve this by mapping the batch records based on the stream name.

      zalmane Please, review.  

       

      Attachments

        Activity

          People

            Unassigned Unassigned
            yossish Yossi Shirizli
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: