Uploaded image for project: 'OpenShift Etcd'
  1. OpenShift Etcd
  2. ETCD-410

Add Controller health to CEO liveness probe

XMLWordPrintable

    • Icon: Story Story
    • Resolution: Done
    • Icon: Undefined Undefined
    • None
    • None
    • None
    • 3
    • False
    • None
    • False
    • ETCD Sprint 234

      We've had several forum cases and bugs already where a restart of the CEO was fixing issues that could be resolved automatically by a liveness probe.

      We previously traced it down to stuck/deadlocked controllers, missing timeouts in grpc calls and other issues we haven't been able to find yet. Since the list of failures that can happen is pretty large, we should add a liveness probe to the CEO that will periodically health check:

      • all controllers have been running sync at least once in the last 5/10 minutes
      • on failure, produce a goroutine dump to analyse what went wrong

      This check should not indicate whether the etcd cluster itself is healthy, it's purely for the CEO itself.

            tjungblu@redhat.com Thomas Jungblut
            tjungblu@redhat.com Thomas Jungblut
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved:

                Estimated:
                Original Estimate - 3 minutes
                3m
                Remaining:
                Remaining Estimate - 3 minutes
                3m
                Logged:
                Time Spent - Not Specified
                Not Specified