Uploaded image for project: 'OpenShift Etcd'
  1. OpenShift Etcd
  2. ETCD-410

Add Controller health to CEO liveness probe

    XMLWordPrintable

Details

    • Story
    • Resolution: Done
    • Undefined
    • None
    • None
    • None
    • 3
    • False
    • None
    • False
    • ETCD Sprint 234

    Description

      We've had several forum cases and bugs already where a restart of the CEO was fixing issues that could be resolved automatically by a liveness probe.

      We previously traced it down to stuck/deadlocked controllers, missing timeouts in grpc calls and other issues we haven't been able to find yet. Since the list of failures that can happen is pretty large, we should add a liveness probe to the CEO that will periodically health check:

      • all controllers have been running sync at least once in the last 5/10 minutes
      • on failure, produce a goroutine dump to analyse what went wrong

      This check should not indicate whether the etcd cluster itself is healthy, it's purely for the CEO itself.

      Attachments

        Issue Links

          Activity

            People

              tjungblu@redhat.com Thomas Jungblut
              tjungblu@redhat.com Thomas Jungblut
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - 3 minutes
                  3m
                  Remaining:
                  Remaining Estimate - 3 minutes
                  3m
                  Logged:
                  Time Spent - Not Specified
                  Not Specified