Loading...

Type: Bug
Resolution: Done
Priority: Major
Fix Version/s: Logging 5.7.0
Affects Version/s: Logging 5.5.7, Logging 5.6.2, Logging 5.7.0
Component/s: Log Storage
Labels:
- devel_ack+

Blocked:
False
Blocked Reason:
None
Ready:
False
Docs QE Status:
NEW
QE Status:
VERIFIED
Release Note Text:

Hide
Before this update, the static LokiStack memberlist configuration using only private IP networds caused the LokiStack pods to crashloop under the condition that the OCP cluster pod network is using a non-private IP network range. With this update, the LokiStack administrator can select to use the pod network for the memberlist configuration and resolves the issue.

Show
Before this update, the static LokiStack memberlist configuration using only private IP networds caused the LokiStack pods to crashloop under the condition that the OCP cluster pod network is using a non-private IP network range. With this update, the LokiStack administrator can select to use the pod network for the memberlist configuration and resolves the issue.
Git Pull Request:
https://github.com/grafana/loki/pull/8672, https://github.com/grafana/loki/pull/8958
Intelligence Requested:
Market:

Sprint:
Log Storage - Sprint 232, Log Storage - Sprint 233, Log Storage - Sprint 234

Test Link:
https://polarion.engineering.redhat.com/polarion/#/project/OSE/workitem?id=OCP-62002
Test Coverage:

?

SFDC Cases Links:
SFDC Cases Counter:
SFDC Cases Open:

Description of problem:

On OpenShift clusters where the administrators choose to use non-private networks (RFC 1918 and RFC 6598) for the cluster pod network, it is not possible to deploy LokiStack cluster. Due to the default assignment in the hash ring configuration to advertise IP address from private IP networks only, the ingester, distributor and querier component are able to form a healthy gossip ring.

Version-Release number of selected component (if applicable):

OpenShift Logging 5.5

OpenShift Logging 5.6

How reproducible:

Always

Steps to Reproduce:

...

Actual results:

Crashlooping pods:

logging-loki-compactor-0                      0/1     CrashLoopBackOff   
logging-loki-distributor-74f676459-9gbzh      0/1     CrashLoopBackOff   
logging-loki-distributor-74f676459-zlqhj      0/1     CrashLoopBackOff   
logging-loki-ingester-0                       0/1     CrashLoopBackOff   
logging-loki-querier-6d56f558b-rnflv          0/1     CrashLoopBackOff   
logging-loki-querier-6d56f558b-z7h8f          0/1     CrashLoopBackOff

Services from pods has no endpoints.
Error message:

failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided

Expected results:

All LokiStack pods required to form a health hash ring advertise IP address from the pod network range. In turn all pods are healthy and normal operations resume.