Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-4884

x509 certificate verification failure on collector while forwarding to Loki on IPv6 cluster

XMLWordPrintable

    • False
    • None
    • False
    • NEW
    • VERIFIED
    • Hide
      Without this update, collector could fail with a x509 certificate verification failure when forwarding logs to Loki on an IPv6 cluster. This update explicitly sets the environment variable KUBERNETES_SERVICE_HOST to 'kubernetes.default.svc' to resolve this issue.
      Show
      Without this update, collector could fail with a x509 certificate verification failure when forwarding logs to Loki on an IPv6 cluster. This update explicitly sets the environment variable KUBERNETES_SERVICE_HOST to 'kubernetes.default.svc' to resolve this issue.
    • Release Note Not Required

      Description:

      x509 certificate verification failure seen on collector pods (vector) when logs are forwarded to default logStore Loki on single stack IPv6 cluster.

      No such error seen on Log 5.8.1 on IPv6 cluster.

      CLO Image: registry.redhat.io/openshift-logging/cluster-logging-rhel9-operator@sha256:91964d8a9c1395fec7b120a7214f5b9b0361d1eb3f09f39ccededda8d78144c6

      Logs:

      2023-12-13T10:28:07.381012Z ERROR kube_client::client::builder: failed with error error trying to connect: error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1889:: hostname mismatch
2023-12-13T10:28:07.381043Z  WARN vector::kubernetes::reflector: Watcher Stream received an error. Retrying. error=InitialListFailed(HyperError(hyper::Error(Connect, ConnectError { error: Error { code: ErrorCode(1), cause: Some(Ssl(ErrorStack([Error
{ code: 167772294, library: "SSL routines", function: "tls_post_process_server_certificate", reason: "certificate verify failed", file: "ssl/statem/statem_clnt.c", line: 1889 }
]))) }, verify_result: X509VerifyResult { code: 62, error: "hostname mismatch" } })))

      How reproducible: Always

      Steps to reproduce:
      1) Deploy CLO and LO v5.9
      2) Forward logs to Loki using vector
      3) Observe collector pod logs

      Actual Result: X509 certificate failure seen on collector pods logs

      Expected Result: Logs should be forwarded to Loki without errors

      Additional Info:

      $ oc get csv
NAME                     DISPLAY                     VERSION   REPLACES   PHASE
cluster-logging.v5.9.0   Red Hat OpenShift Logging   5.9.0                Succeeded
loki-operator.v5.9.0     Loki Operator               5.9.0                Succeeded

      LokiStack:

      apiVersion: loki.grafana.com/v1
kind: LokiStack
metadata:
  name: lokistack-sample
spec:
  hashRing:
    memberlist:
      enableIPv6: true
    type: memberlist
  managementState: Managed
  size: 1x.demo
  storage:
    secret:
      name: s3-secret
      type: s3
  storageClassName: nfs
  tenants:
    mode: openshift-logging
  rules:
    enabled: true
    namespaceSelector:
      matchLabels:
        openshift.io/cluster-monitoring: "true"
    selector:
      matchLabels:
        openshift.io/cluster-monitoring: "true"

      CLF:

      apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: instance
  namespace: openshift-logging
spec:
  pipelines:
    - inputRefs:
        - application
      name: all-logs-to-lokistack
      outputRefs:
        - default
status:
  conditions:
    - lastTransitionTime: '2023-12-13T10:33:48Z'
      status: 'True'
      type: Ready

      vector.toml attached

        1. ipv6-vector (2).toml
          7 kB

              vparfono Vitalii Parfonov
              rhn-support-kbharti Kabir Bharti
              Kabir Bharti Kabir Bharti
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: