Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-6987

Change Loki configuration to update memberlist ring when ingester becomes unhealthy

XMLWordPrintable

    • 2
    • False
    • Hide

      None

      Show
      None
    • False
    • NEW
    • VERIFIED
    • Hide
      Before this update, Loki ingesters that got into an unhealthy state due to networking issues stayed in that state even when the network recovered. With this update, the Loki Operator adds a configuration option that configures Loki to do service discovery more often, which makes unhealthy ingesters rejoin the group.
      Show
      Before this update, Loki ingesters that got into an unhealthy state due to networking issues stayed in that state even when the network recovered. With this update, the Loki Operator adds a configuration option that configures Loki to do service discovery more often, which makes unhealthy ingesters rejoin the group.
    • Release Note Not Required
    • Log Storage - Sprint 269, Log Storage - Sprint 270, Log Storage - Sprint 271

      Description of problem:

      LOG-6885 and LOG-6968 show that there are currently issues when ingesters turn unhealthy and network connection is not restored quickly enough.

      This issue aims to provide a workaround for these issues by enabling options in the Loki configuration that will cause it to remove unhealthy ingesters from the memberlist ring immediately, so that a rebalancing of streams happens. It also enables an option to periodically rejoin the ring using the existing discovery mechanism, so that ingesters having network issues can quickly rejoin once the issues are gone.

      Version-Release number of selected component (if applicable):

      6.2.0

      How reproducible:

      See reproduction steps in LOG-6968.

      Additional info:

      This issue tracks a workaround for the issue with heartbeat messages described in LOG-6968. This change might be rolled back in-part or completely once the issue is fixed in LOG-6968.

              rojacob@redhat.com Robert Jacob
              rojacob@redhat.com Robert Jacob
              Kabir Bharti Kabir Bharti
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: