Uploaded image for project: 'Red Hat Fuse'
  1. Red Hat Fuse
  2. ENTESB-7537

Implement Camel Aws-Kinesis Using Kinesis Client Library(KCL)

XMLWordPrintable

    • Icon: Feature Feature
    • Resolution: Won't Do
    • Icon: Major Major
    • None
    • jboss-fuse-6.3
    • Camel
    • Todo

      Amazon provides 2 options for Kinesis Java Client:

      • Kinesis Client Library(KCL)
      • Kinesis Streams API

      "Kinesis Streams API" is a low level API set, and "KCL" is a rich library.
      In the official document, "KCL" is described like this:

      The KCL is compiled into your application to enable fault-tolerant consumption of data from the stream.
      The KCL ensures that for every shard there is a record processor running and processing that shard.
      The library also simplifies reading data from the stream.
      The KCL uses an Amazon DynamoDB table to store control data. It creates one table per application that is processing data.

      The KCL takes care of many of the complex tasks associated with distributed computing, such as load-balancing across multiple instances, responding to instance failures, checkpointing processed records, and reacting to resharding.
      The KCL enables you to focus on writing record processing logic.

      Since "Kinesis Streams API" is used to implement current aws-kinesis component, usability and scalability are not provided by default.

      • User needs to specify shardID(ENTESB-7517) which might be changed dynamically
      • If a Stream uses multiple shards, user needs to create multiple endpoints for it. (Increasing number of shards are tips to scale out. If there are 200 shards, user needs to maintain 200 endpoints)
      • If user wants to consume a same shard from multiple Fuse instances, user needs to implement load-balancing

      By using "KCL" to implement aws-kinesis, above current difficulties/concern can be resolved.

              acosenti Andrea Cosentino
              rhn-support-hfuruich Hisao Furuichi
              Votes:
              7 Vote for this issue
              Watchers:
              10 Start watching this issue

                Created:
                Updated:
                Resolved: