Uploaded image for project: 'AMQ Streams'
  1. AMQ Streams
  2. ENTMQST-4286

[DOC] Inaccurate information is described in "5.2. Data durability"

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Undefined Undefined
    • 2.2.0.GA
    • 2.2.0.GA
    • documentation
    • None
    • False
    • None
    • False
    • If Release Note Needed, Set a Value
    • Set a Value

      Inaccurate information is described in "5.2. Data durability" (https://access.redhat.com/documentation/en-us/red_hat_amq_streams/2.2/html-single/kafka_configuration_tuning/index#data_durability)

      acks=all forces a partition leader to replicate messages to a certain number of followers before acknowledging that the message request was successfully received.

      This is not wrong, but "leader will wait for the full set of in-sync replicas to acknowledge"[Kafka doc acks=all]

      The acks=all setting offers the strongest guarantee of delivery, but it will increase the latency between the producer sending a message and receiving acknowledgment. If you don’t require such strong guarantees, a setting of acks=0 or acks=1 provides either no delivery guarantees or only acknowledgment that the leader partition has written the record to its log.

      I don't see any problem.

      The number of brokers which need to have appended the messages to their logs before the acknowledgment is sent to the producer is determined by the topic’s min.insync.replicas configuration.

      This is not accurate, not "min.insync.replicas", but "the full set of in-sync replicas(ISR)"[Kafka doc acks=all].

      • For example, in the case of acks=all, replicas=3 and min.insync.replicas=2:
        • when ISR=3 (normal status), the leader will wait for 2 follower replica broker's acks.
        • when ISR=2 (one follower is down or delayed), the leader will wait for only one normal follower replica broker's ack.
        • when ISR=1 (all the followers are down or delayed), the producer will raise an exception (either NotEnoughReplicas or NotEnoughReplicasAfterAppend) because of "min.insync.replicas".

      A typical starting point is to have a topic replication factor of 3, with two in-sync replicas on other brokers.

      I could not understand this sentence. Does this mean that "A typical scenario would be to create a topic with a replication factor of 3, set min.insync.replicas to 2, and produce with acks of "all" "[Kafka doc min.insync.replicas] ?

      Was "two min.insync.replicas" intended instead of "two in-sync replicas(ISR)"?

       

      In this configuration, the producer can continue unaffected if a single broker is unavailable. If a second broker becomes unavailable, the producer won’t receive acknowledgments and won’t be able to produce more messages.

      This may not be wrong, but I think "unaffected" could be misleading. In the case of acks=all, the leader will wait for "the full set of in-sync replicas", not "min.insync.replicas". So in particular, when a failure occurs in a broker(leader/follower) the producer will be in a delay until it is detected and removed from the ISR. More specifically a producer connects to the leader replica, so when the leader replica is down, it will takes about "zookeeper.session.timeout.ms(default: 18 seconds)" to be replaced to new one. And even a follower replica is down it will takes about "zookeeper.session.timeout.ms(default: 18 seconds)" or "replica.lag.time.max.ms(defalt : 30 seconds)" to be removed from ISR.

      Although after a failure, if a new leader replica is selected and the number of ISRs decreases, the process will continue with decreased ISR, but the delay from failure to recovery may not be negligible that can be categorized as "unaffected".
       
      [References]

      • [Kafka doc acks=all] https://kafka.apache.org/documentation/
        • This means the leader will wait for the full set of in-sync replicas to acknowledge the record. This guarantees that the record will not be lost as long as at least one in-sync replica remains alive. This is the strongest available guarantee. This is equivalent to the acks=-1 setting.
      • [Kafka doc min.insync.replicas] https://kafka.apache.org/documentation/
        • When a producer sets acks to "all" (or "-1"), min.insync.replicas specifies the minimum number of replicas that must acknowledge a write for the write to be considered successful. If this minimum cannot be met, then the producer will raise an exception (either NotEnoughReplicas or NotEnoughReplicasAfterAppend).
          When used together, min.insync.replicas and acks allow you to enforce greater durability guarantees. A typical scenario would be to create a topic with a replication factor of 3, set min.insync.replicas to 2, and produce with acks of "all". This will ensure that the producer raises an exception if a majority of replicas do not receive a write.

       

            pmellor@redhat.com Paul Mellor
            rhn-support-tyamashi Tomonari Yamashita
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: