Uploaded image for project: 'AMQ Streams'
  1. AMQ Streams
  2. ENTMQST-4521

Connector auto-restart counter does not reset back to 0

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Undefined Undefined
    • 2.4.0.GA
    • None
    • None
    • None

      When auto-restarting connectors, we want to avoid trying to restart a failing connector again and again. So we use a counter with a timestamp to do a limited backoff. The connector is restarted only up to 7 times and each restart is done later after the previous one. (E.g. first restart is immediate, the second restart will not be done sooner than 2 minutes after the first, the third will be done at least 6 minutes after the second, etc.) The latest state is tracked in the status section of the KafkaConnector CR - it stores the number of restarts already done and the timestamp of the last restart.

      This works as designed. If your connector is broken and fails all the time, we will not keep restarting it forever and force you to deal with it. But also, if you have a longer issue - e.g. a 10-minute network outage - the backoff will try to restart it not only immediately but also a bit later.

      However, there is currently no reset to the tracked restart state. If your connector fails for example, once per week but works fine in between, it will still accumulate the restarts in its status. And after 7 weeks, it will reach the maximal count of 7 restarts and will not be considered for restarts anymore.

      We need to find a middle ground - a mechanism that after some time decides the connector is fixed and resets the restart counter. That way, the next restart sequence will start from scratch. That way, the auto-restart can be useful even for longer time periods. For example, if the connector is running fine for N minutes, the operator would reset the counter to 0. The time might be configurable in the auto-restart section in .spec to handle different requirements for different situations.

      Created by Strimzi#7891

              Unassigned Unassigned
              scholzj JAkub Scholz
              Jan Kalinic Jan Kalinic
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: