Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-2130

Vector - Collector pods fails to start when forwarding logs to Loki using tenantKey

    XMLWordPrintable

Details

    • Logging (Core) - Sprint 213

    Description

      When forwarding logs to Loki using teneantKey, the Vector Collector pods fails to start due to configuration error 'error=expected an equals, found a right brace at line 59 column 33'

      Server Version: 4.10.0-0.nightly-2022-01-11-065245
      Kubernetes Version: v1.22.1+6859754

      Steps to reproduce the issue:

      1 Deploy Logging 5.4

      $ oc get csv
      NAME                            DISPLAY                            VERSION   REPLACES   PHASE
      clusterlogging.v5.4.0           Red Hat OpenShift Logging          5.4.0                Succeeded
      elasticsearch-operator.v5.4.0   OpenShift Elasticsearch Operator   5.4.0                Succeeded 

      2 Create a CLO instance.

      apiVersion: "logging.openshift.io/v1"
      kind: "ClusterLogging"
      metadata:
        name: "instance" 
        namespace: "openshift-logging"
      spec:
        managementState: "Managed"  
        logStore:
          type: "elasticsearch"  
          retentionPolicy: 
            application:
              maxAge: 10h
            infra:
              maxAge: 10h
            audit:
              maxAge: 10h
          elasticsearch:
            nodeCount: 1 
            storage: {} 
            resources: 
                limits:
                  memory: "4Gi"
                requests:
                  memory: "1Gi"
            proxy: 
              resources:
                limits:
                  memory: 256Mi
                requests:
                  memory: 256Mi
            redundancyPolicy: "ZeroRedundancy"
        visualization:
          type: "kibana"  
          kibana:
            replicas: 1
        collection:
          logs:
            type: "vector"  
            vector: {} 

      3 Create a CLF instance to forward logs to Loki.

      apiVersion: logging.openshift.io/v1
      kind: ClusterLogForwarder
      metadata:
        name: instance
        namespace: openshift-logging
      spec:
        outputs:
        - name: loki-server
          type: loki
          loki:
            tenantKey: kubernetes.pod_name
          url: http://loki-ingester-http-lokistack-dev.lokicluster.svc:3100
        pipelines:
        - name: to-loki
          inputRefs:
          - application
          outputRefs:
          - loki-server 

      4 Check the collector pods status and logs.

      $ oc get pods
      NAME                                           READY   STATUS             RESTARTS      AGE
      cluster-logging-operator-5cb5867fc7-5rbtm      1/1     Running            0             5h12m
      collector-4c7bc                                1/2     CrashLoopBackOff   1 (16s ago)   19s
      collector-d6mp9                                1/2     CrashLoopBackOff   1 (15s ago)   19s
      collector-f2h56                                1/2     CrashLoopBackOff   1 (16s ago)   19s
      collector-h9mnt                                1/2     CrashLoopBackOff   1 (14s ago)   19s
      collector-jnb2w                                1/2     Error              2 (15s ago)   19s
      collector-l4v4t                                1/2     CrashLoopBackOff   1 (16s ago)   19s
      elasticsearch-cdm-ib58ker0-1-6bf8d674b-m2zr9   2/2     Running            0             45s
      kibana-6cc8d5f7bf-6fp6c                        2/2     Running            0             45s
      
      $ oc logs collector-h9mnt -c collector
      Jan 13 09:18:39.048  INFO vector::app: Log level is enabled. level="debug"
      Jan 13 09:18:39.048  INFO vector::sources::host_metrics: PROCFS_ROOT is unset. Using default '/proc' for procfs root.
      Jan 13 09:18:39.048  INFO vector::sources::host_metrics: SYSFS_ROOT is unset. Using default '/sys' for sysfs root.
      Jan 13 09:18:39.048  INFO vector::app: Loading configs. path=[("/etc/vector/vector.toml", Some(Toml))]
      Jan 13 09:18:39.051 ERROR vector::cli: Configuration error. error=expected an equals, found a right brace at line 59 column 33
       

      Generated vector.toml file:

      # Logs from containers (including openshift containers)
      [sources.raw_container_logs]
      type = "kubernetes_logs"
      auto_partial_merge = true
      exclude_paths_glob_patterns = ["/var/log/pods/openshift-logging_collector-*/*/*.log", "/var/log/pods/openshift-logging_elasticsearch-*/*/*.log", "/var/log/pods/openshift-logging_kibana-*/*/*.log"][transforms.container_logs]
      type = "remap"
      inputs = ["raw_container_logs"]
      source = '''
        level = "unknown"
        if match(.message,r'(Warning|WARN|W[0-9]+|level=warn|Value:warn|"level":"warn")'){
          level = "warn"
        } else if match(.message, r'Info|INFO|I[0-9]+|level=info|Value:info|"level":"info"'){
          level = "info"
        } else if match(.message, r'Error|ERROR|E[0-9]+|level=error|Value:error|"level":"error"'){
          level = "error"
        } else if match(.message, r'Debug|DEBUG|D[0-9]+|level=debug|Value:debug|"level":"debug"'){
          level = "debug"
        }
        .level = level  .pipeline_metadata.collector.name = "vector"
        .pipeline_metadata.collector.version = "0.14.1"
        ip4, err = get_env_var("NODE_IPV4")
        .pipeline_metadata.collector.ipaddr4 = ip4
        received, err = format_timestamp(now(),"%+")
        .pipeline_metadata.collector.received_at = received
        .pipeline_metadata.collector.error = err
       '''
      [transforms.route_container_logs]
      type = "route"
      inputs = ["container_logs"]
      route.app = '!(starts_with!(.kubernetes.pod_namespace,"kube") && starts_with!(.kubernetes.pod_namespace,"openshift") && .kubernetes.pod_namespace == "default")'
      # Rename log stream to "application"
      [transforms.application]
      type = "remap"
      inputs = ["route_container_logs.app"]
      source = """
      .log_type = "application"
      """
      [transforms.to-loki]
      type = "remap"
      inputs = ["application"]
      source = """
      .
      """[sinks.loki_server]
      type = "loki"
      inputs = ["to-loki"]
      endpoint = "http://loki-ingester-http-lokistack-dev.lokicluster.svc:3100"
      tenant_id = {kubernetes.pod_name}[sinks.loki_server.encoding]
      codec = "json"[sinks.loki_server.labels]
      kubernetes_container_name = "{{kubernetes.container_name}}"
      kubernetes_host = "${NODE_NAME}"
      kubernetes_namespace_name = "{{kubernetes.namespace_name}}"
      kubernetes_pod_name = "{{kubernetes.pod_name}}"
      log_type = "{{log_type}}" 

      Attachments

        Activity

          People

            vimalkum@redhat.com Vimal Kumar
            rhn-support-ikanse Ishwar Kanse
            Ishwar Kanse Ishwar Kanse
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: