Loading...

Type: Bug
Resolution: Done-Errata
Priority: Normal
Fix Version/s: Logging 5.8.0
Affects Version/s: Logging 5.5.0
Component/s: Log Storage, Loki
Labels:
- devel_ack+

Blocked:
False
Blocked Reason:
None
Ready:
False
Docs QE Status:
NEW
QE Status:
VERIFIED
Release Note Text:

Hide
Before this update, deploying LokiStack on IPv6-only or dual-stack OCP clusters caused the LokiStack memberlist registration to fail and distributor pods into a crashloop. With this update, the administrator can enable IPv6 using `lokistack.spec.hashRing.memberlist.enableIPv6: true` resolves the issue and the LokiStack components can be operated on the aforementioned OCP cluster types.

Show
Before this update, deploying LokiStack on IPv6-only or dual-stack OCP clusters caused the LokiStack memberlist registration to fail and distributor pods into a crashloop. With this update, the administrator can enable IPv6 using `lokistack.spec.hashRing.memberlist.enableIPv6: true` resolves the issue and the LokiStack components can be operated on the aforementioned OCP cluster types.
Release Note Type:
Bug Fix

Sprint:
Log Storage - Sprint 221, Log Storage - Sprint 222, Log Storage - Sprint 223, Log Storage - Sprint 224, Log Storage - Sprint 226, Log Storage - Sprint 227, Log Storage - Sprint 228, Log Storage - Sprint 229, Log Storage - Sprint 231, Log Storage - Sprint 232, Log Storage - Sprint 235, Log Storage - Sprint 236, Log Storage - Sprint 237, Log Storage - Sprint 242, Log Storage - Sprint 243

SFDC Cases Links:
SFDC Cases Counter:
SFDC Cases Open:

transport: authentication handshake failed: x509, certificate is invalid to lokistack-sample-ingester-grpc.openshift-logging.svc.cluster.local

level=warn ts=2022-07-08T03:18:27.099657374Z caller=logging.go:72 orgID=application msg="POST /loki/api/v1/push (500) 2.495076ms Response: \"rpc error: code = Unavailable desc = connection error: desc = \\\"transport: authentication handshake failed: x509: certificate is valid for .lokistack-sample-distributor-grpc.openshift-logging.svc, *.lokistack-sample-distributor-grpc.openshift-logging.svc.cluster.local, lokistack-sample-distributor-grpc.openshift-logging.svc, lokistack-sample-distributor-grpc.openshift-logging.svc.cluster.local, not lokistack-sample-ingester-grpc.openshift-logging.svc.cluster.local\\\"\\n\" ws: false; Accept: */; Accept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3; Content-Length: 13290; Content-Type: application/json; User-Agent: Ruby; X-Forwarded-For: fd01:0:0:5::27; X-Forwarded-Prefix: /api/logs/v1/application; X-Scope-Orgid: application; "

Step to reproduce:

Deploy lokistack on IPv6 cluster.
fluentd forward logs to lokistack.
note: workaround fluentd IPv6 bugs by updating bind "# {ENV['POD_IP']}" to bind "#{ENV['POD_IP']}
" in fluentd.conf in Unmanagement status
Check the collector pod logs or distributor

Actual result:
$oc logs lokistack-sample-distributor-fbff69857-2gxnh
......

level=warn ts=2022-07-08T03:27:34.697142051Z caller=logging.go:72 orgID=application msg="POST /loki/api/v1/push (500) 2.010046ms Response: \"rpc error: code = Unavailable desc = connection error: desc = \\\"transport: authentication handshake failed: x509: certificate is valid for *.lokistack-sample-distributor-grpc.openshift-logging.svc, *.lokistack-sample-distributor-grpc.openshift-logging.svc.cluster.local, lokistack-sample-distributor-grpc.openshift-logging.svc, lokistack-sample-distributor-grpc.openshift-logging.svc.cluster.local, not lokistack-sample-ingester-grpc.openshift-logging.svc.cluster.local\\\"\\n\" ws: false; Accept: */*; Accept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3; Content-Length: 15397; Content-Type: application/json; User-Agent: Ruby; X-Forwarded-For: fd01:0:0:4::34; X-Forwarded-Prefix: /api/logs/v1/application; X-Scope-Orgid: application; "
level=warn ts=2022-07-08T03:27:35.918733473Z caller=pool.go:184 msg="removing ingester failing healthcheck" addr=127.0.0.1:9095 reason="rpc error: code = Unavailable desc = connection error: desc = \"transport: authentication handshake failed: x509: certificate is valid for *.lokistack-sample-distributor-grpc.openshift-logging.svc, *.lokistack-sample-distributor-grpc.openshift-logging.svc.cluster.local, lokistack-sample-distributor-grpc.openshift-logging.svc, lokistack-sample-distributor-grpc.openshift-logging.svc.cluster.local, not lokistack-sample-ingester-grpc.openshift-logging.svc.cluster.local\""
level=warn ts=2022-07-08T03:28:10.307325601Z caller=logging.go:72 orgID=application msg="POST /loki/api/v1/push (500) 2.022913ms Response: \"rpc error: code = Unavailable desc = connection error: desc = \\\"transport: authentication handshake failed: x509: certificate is valid for *.lokistack-sample-distributor-grpc.openshift-logging.svc, *.lokistack-sample-distributor-grpc.openshift-logging.svc.cluster.local, lokistack-sample-distributor-grpc.openshift-logging.svc, lokistack-sample-distributor-grpc.openshift-logging.svc.cluster.local, not lokistack-sample-ingester-grpc.openshift-logging.svc.cluster.local\\\"\\n\" ws: false; Accept: */*; Accept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3; Content-Length: 13290; Content-Type: application/json; User-Agent: Ruby; X-Forwarded-For: fd01:0:0:5::27; X-Forwarded-Prefix: /api/logs/v1/application; X-Scope-Orgid: application; "