Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-4570

[release-5.6] transport: authentication handshake failed: x509 on IPv6 Cluster

    XMLWordPrintable

Details

    • False
    • None
    • False
    • NEW
    • VERIFIED
    • Hide
      Before this update, deploying LokiStack on IPv6-only or dual-stack OCP clusters caused the LokiStack memberlist registration to fail and distributor pods into a crashloop. With this update, the administrator can enable IPv6 using `lokistack.spec.hashRing.memberlist.enableIPv6: true` resolves the issue and the LokiStack components can be operated on the aforementioned OCP cluster types.
      Show
      Before this update, deploying LokiStack on IPv6-only or dual-stack OCP clusters caused the LokiStack memberlist registration to fail and distributor pods into a crashloop. With this update, the administrator can enable IPv6 using `lokistack.spec.hashRing.memberlist.enableIPv6: true` resolves the issue and the LokiStack components can be operated on the aforementioned OCP cluster types.
    • Bug Fix
    • Log Storage - Sprint 243

    Description

      transport: authentication handshake failed: x509, certificate is invalid to lokistack-sample-ingester-grpc.openshift-logging.svc.cluster.local

      level=warn ts=2022-07-08T03:18:27.099657374Z caller=logging.go:72 orgID=application msg="POST /loki/api/v1/push (500) 2.495076ms Response: \"rpc error: code = Unavailable desc = connection error: desc = \\\"transport: authentication handshake failed: x509: certificate is valid for .lokistack-sample-distributor-grpc.openshift-logging.svc, *.lokistack-sample-distributor-grpc.openshift-logging.svc.cluster.local, lokistack-sample-distributor-grpc.openshift-logging.svc, lokistack-sample-distributor-grpc.openshift-logging.svc.cluster.local, not lokistack-sample-ingester-grpc.openshift-logging.svc.cluster.local\\\"\\n\" ws: false; Accept: */; Accept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3; Content-Length: 13290; Content-Type: application/json; User-Agent: Ruby; X-Forwarded-For: fd01:0:0:5::27; X-Forwarded-Prefix: /api/logs/v1/application; X-Scope-Orgid: application; "

      Step to reproduce:

      • Deploy lokistack on IPv6 cluster.
      • fluentd forward logs to lokistack.
        note: workaround fluentd IPv6 bugs by updating bind "# {ENV['POD_IP']}" to bind "#{ENV['POD_IP']}
        " in fluentd.conf in Unmanagement status
        Check the collector pod logs or distributor

      Actual result:
      $oc logs lokistack-sample-distributor-fbff69857-2gxnh
      ......

      level=warn ts=2022-07-08T03:27:34.697142051Z caller=logging.go:72 orgID=application msg="POST /loki/api/v1/push (500) 2.010046ms Response: \"rpc error: code = Unavailable desc = connection error: desc = \\\"transport: authentication handshake failed: x509: certificate is valid for *.lokistack-sample-distributor-grpc.openshift-logging.svc, *.lokistack-sample-distributor-grpc.openshift-logging.svc.cluster.local, lokistack-sample-distributor-grpc.openshift-logging.svc, lokistack-sample-distributor-grpc.openshift-logging.svc.cluster.local, not lokistack-sample-ingester-grpc.openshift-logging.svc.cluster.local\\\"\\n\" ws: false; Accept: */*; Accept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3; Content-Length: 15397; Content-Type: application/json; User-Agent: Ruby; X-Forwarded-For: fd01:0:0:4::34; X-Forwarded-Prefix: /api/logs/v1/application; X-Scope-Orgid: application; "
      level=warn ts=2022-07-08T03:27:35.918733473Z caller=pool.go:184 msg="removing ingester failing healthcheck" addr=127.0.0.1:9095 reason="rpc error: code = Unavailable desc = connection error: desc = \"transport: authentication handshake failed: x509: certificate is valid for *.lokistack-sample-distributor-grpc.openshift-logging.svc, *.lokistack-sample-distributor-grpc.openshift-logging.svc.cluster.local, lokistack-sample-distributor-grpc.openshift-logging.svc, lokistack-sample-distributor-grpc.openshift-logging.svc.cluster.local, not lokistack-sample-ingester-grpc.openshift-logging.svc.cluster.local\""
      level=warn ts=2022-07-08T03:28:10.307325601Z caller=logging.go:72 orgID=application msg="POST /loki/api/v1/push (500) 2.022913ms Response: \"rpc error: code = Unavailable desc = connection error: desc = \\\"transport: authentication handshake failed: x509: certificate is valid for *.lokistack-sample-distributor-grpc.openshift-logging.svc, *.lokistack-sample-distributor-grpc.openshift-logging.svc.cluster.local, lokistack-sample-distributor-grpc.openshift-logging.svc, lokistack-sample-distributor-grpc.openshift-logging.svc.cluster.local, not lokistack-sample-ingester-grpc.openshift-logging.svc.cluster.local\\\"\\n\" ws: false; Accept: */*; Accept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3; Content-Length: 13290; Content-Type: application/json; User-Agent: Ruby; X-Forwarded-For: fd01:0:0:5::27; X-Forwarded-Prefix: /api/logs/v1/application; X-Scope-Orgid: application; "
      

       

      Attachments

        Activity

          People

            ptsiraki@redhat.com Periklis Tsirakidis
            anli@redhat.com Anping Li
            Kabir Bharti Kabir Bharti
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: