Details
-
Enhancement
-
Resolution: Unresolved
-
Major
-
None
-
None
-
False
-
None
-
False
Description
We used to have the batch implemented via a transaction but changed it to pipeline as in multi sharded deployment of the Redis DB, keys aimed to different shards can't be part of the same transaction.
With pipelines we can end up with having duplicate records (although we reduced dramatically the chances to get this as part of the DBZ-4966 fix) not to mention we should parse the responses one by one as some of the commands might fail (while transactions can be rollbacked and once it succeeds, all of its commands can be marked as processed).
We can switch back to transactions by having a transaction per stream (stream name is the key on which the hashtag function is performed tunneled it to the same shard) and guarantee the whole transaction will go to the same shard.
We will achieve this by mapping the batch records based on the stream name.
zalmane Please, review.