Description of problem:

LOG-6885 and LOG-6968 show that there are currently issues when ingesters turn unhealthy and network connection is not restored quickly enough.

This issue aims to provide a workaround for these issues by enabling options in the Loki configuration that will cause it to remove unhealthy ingesters from the memberlist ring immediately, so that a rebalancing of streams happens. It also enables an option to periodically rejoin the ring using the existing discovery mechanism, so that ingesters having network issues can quickly rejoin once the issues are gone.

Version-Release number of selected component (if applicable):

6.2.0

How reproducible:

See reproduction steps in LOG-6968.

Additional info:

This issue tracks a workaround for the issue with heartbeat messages described in LOG-6968. This change might be rolled back in-part or completely once the issue is fixed in LOG-6968.