-
Bug
-
Resolution: Unresolved
-
Critical
-
None
-
Logging 5.8.4, Logging 5.8.5, Logging 5.9.0
-
False
-
None
-
False
-
NEW
-
NEW
-
Bug Fix
-
-
Description of problem:
Unhealthy ingesters are not leaving the hash ring. When replication_factor is set to 2, and an unhealthy member is selected, causing log forwarding to fail.
$ oc exec -n openshift-logging logging-loki-ingester-0 -- curl -k --cert /var/run/tls/http/server/tls.crt --key /var/run/tls/http/server/tls.key https://localhost:3100/ring -H "Accept: application/json" | jq | grep -B1 state % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 17117 0 17117 0 0 491k 0 --:--:-- --:--:-- --:--:-- 491k "id": "logging-loki-ingester-0", "state": "ACTIVE", -- "id": "logging-loki-ingester-1", "state": "ACTIVE", -- "id": "logging-loki-ingester-2", "state": "UNHEALTHY",
Version-Release number of selected component (if applicable):
Logging 5.8.4, 5.8.5, 5.9.0
How reproducible:
Always
Steps to Reproduce:
- Install OpenShift Logging with LokiStack
- Shut down the node where ingester is deployed
- Confirm if logs are being collected in the web console
Actual results:
Unhealthy ingesters are leaving the hash ring
Expected results:
Unhealthy ingestors continue to remain in the hash ring
Additional info:
The following issue could be related:
https://github.com/grafana/loki/issues/8615
