A common but not immediately obvious way to misconfigure LokiStack is to provide it with a secret that is syntactically correct but invalid for accessing the object storage.

This error will only become visible, once Loki tries to interact with the object storage, for example when the ingester tries to push new chunks into it.

A once valid secret might also become invalid over time if it expires or the account is deactivated.

Loki 3.1 now added a metric showing the number of failed chunk flushes: loki_ingester_chunks_flush_failures_total

We should be able to create an alert based on that metric that fires when there are flush failures and alerts the operator if this issue persists for a while indicating a failure with either the object storage itself or the secret.