Uploaded image for project: 'Managed Service - Streams'
  1. Managed Service - Streams
  2. MGDSTRM-9146

Kafka Instance API (admin server) does not return latest from all partitions

XMLWordPrintable

    • False
    • None
    • False
    • No
    • Not Required
    • ---
    • ---
    • MK - Sprint 223, MK - Sprint 224

      Currently, when the front-end sends only the `limit` parameter, the admin server attempts to consume at most `limit` records from the Kafka broker(s). The starting offset for the consumer is (for each partition) calculated as `endOffset` - `limit`. When message reside is more than one partition, the messages from partitions other than the first may not appear in the result set, even if they are newer than some messages from the earlier/lower partitions.

      Without consuming the messages, it is not known which partition holds the latest messages or how the latest messages are distributed across the partitions.

      Possible solutions:

      1. Calculate the starting offset as
        endOffset - (limit / partitionCount)

        . This may result in fewer than `limit` records returned.

      2. Fetch up to `partitionCount` * `limit` records and return up to `limit` records that are chronologically the latest.

              srbiswas@redhat.com Srijoni Biswas
              medgar@redhat.com Michael Edgar
              Kafka Fleet Services
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated:
                Resolved: