Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-8265

journal processing loops after journal offset reset

XMLWordPrintable

    • False
    • None
    • False
    • Critical

      The ibmi journals are logged afainst receivers

      each receiver has a range of offsets

      the system call we make doesn't work properly when the end offset is lower than the maximum in the receiver range

      i.e.

      reciver           offset

      receiver1 1 ->10

      receiver2 11->20

      receiver3 1 -> 10

       

      asking for receiver1 offset 1 to reciver3 offset 10 will only get the first 10 entries from receiver1 and none from receiver2 or receiver3

      we have code to only make requests against incrementing offsets and then move to the new receiver and start at the beginning

      i.e. making two calls:

      receiver1 offset 1 to receiver2 offset 20

      receiver3 ofset 1 to receiver3 offset 10

       

      We also limit the range we ask for as asking for too much can take a long time and we might timeout.

       

      A bug in the code that tries to calculate the end offset returns the wrong range for the receiver when we land exactly in the last receiver. A safety check verifies against the wrong offset and also subtracts the last range twice.

       

      Chances of occuring depend on the frequency of the resets and we default to a range of 1,000,000 so I'd expect this to be rare but it's happened and is quite catastrophic when it does as we then end up looping over the same entries generating a lot of traffic.

            Unassigned Unassigned
            msillencefnz Martin Sillence
            Martin Sillence
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: