Context

Loki Ingesters hold a significant buffer of unflushed data in RAM and participate in a strict quorum-based hash ring. Because of this we cannot simply scale down it's StatefulSet as the application will only get a SIGTERM from Kubernetes. Instead what should happen is the following:

• We should set the ingester to LEAVING in the memberlist righ
• We should call /ingester/shutdown?flush=true to force a flush of the ingester
• Wait for the flush to complete
• Lower the number of Ingester Replicas in the StatefulSet

With this it's clear that the operator will need to interact with the ingester instance. It's not something we currently do so that will also be something to work on

Acceptance Criteria

• We should have a mechanism to interact with the Loki components from the operator code
• We should be able to increase & decrease the number of ingesters without losing data