We've had several forum cases and bugs already where a restart of the CEO was fixing issues that could be resolved automatically by a liveness probe.

We previously traced it down to stuck/deadlocked controllers, missing timeouts in grpc calls and other issues we haven't been able to find yet. Since the list of failures that can happen is pretty large, we should add a liveness probe to the CEO that will periodically health check:

• all controllers have been running sync at least once in the last 5/10 minutes
• on failure, produce a goroutine dump to analyse what went wrong

This check should not indicate whether the etcd cluster itself is healthy, it's purely for the CEO itself.