-
Bug
-
Resolution: Done
-
Critical
-
Logging 5.4.0
-
False
-
False
-
NEW
-
OBSDA-108 - Distribute an alternate Vector Log Collector
-
VERIFIED
-
-
Logging (Core) - Sprint 213
When forwarding logs to Loki using teneantKey, the Vector Collector pods fails to start due to configuration error 'error=expected an equals, found a right brace at line 59 column 33'
Server Version: 4.10.0-0.nightly-2022-01-11-065245
Kubernetes Version: v1.22.1+6859754
Steps to reproduce the issue:
1 Deploy Logging 5.4
$ oc get csv
NAME DISPLAY VERSION REPLACES PHASE
clusterlogging.v5.4.0 Red Hat OpenShift Logging 5.4.0 Succeeded
elasticsearch-operator.v5.4.0 OpenShift Elasticsearch Operator 5.4.0 Succeeded
2 Create a CLO instance.
apiVersion: "logging.openshift.io/v1" kind: "ClusterLogging" metadata: name: "instance" namespace: "openshift-logging" spec: managementState: "Managed" logStore: type: "elasticsearch" retentionPolicy: application: maxAge: 10h infra: maxAge: 10h audit: maxAge: 10h elasticsearch: nodeCount: 1 storage: {} resources: limits: memory: "4Gi" requests: memory: "1Gi" proxy: resources: limits: memory: 256Mi requests: memory: 256Mi redundancyPolicy: "ZeroRedundancy" visualization: type: "kibana" kibana: replicas: 1 collection: logs: type: "vector" vector: {}
3 Create a CLF instance to forward logs to Loki.
apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
name: instance
namespace: openshift-logging
spec:
outputs:
- name: loki-server
type: loki
loki:
tenantKey: kubernetes.pod_name
url: http://loki-ingester-http-lokistack-dev.lokicluster.svc:3100
pipelines:
- name: to-loki
inputRefs:
- application
outputRefs:
- loki-server
4 Check the collector pods status and logs.
$ oc get pods NAME READY STATUS RESTARTS AGE cluster-logging-operator-5cb5867fc7-5rbtm 1/1 Running 0 5h12m collector-4c7bc 1/2 CrashLoopBackOff 1 (16s ago) 19s collector-d6mp9 1/2 CrashLoopBackOff 1 (15s ago) 19s collector-f2h56 1/2 CrashLoopBackOff 1 (16s ago) 19s collector-h9mnt 1/2 CrashLoopBackOff 1 (14s ago) 19s collector-jnb2w 1/2 Error 2 (15s ago) 19s collector-l4v4t 1/2 CrashLoopBackOff 1 (16s ago) 19s elasticsearch-cdm-ib58ker0-1-6bf8d674b-m2zr9 2/2 Running 0 45s kibana-6cc8d5f7bf-6fp6c 2/2 Running 0 45s $ oc logs collector-h9mnt -c collector Jan 13 09:18:39.048 INFO vector::app: Log level is enabled. level="debug" Jan 13 09:18:39.048 INFO vector::sources::host_metrics: PROCFS_ROOT is unset. Using default '/proc' for procfs root. Jan 13 09:18:39.048 INFO vector::sources::host_metrics: SYSFS_ROOT is unset. Using default '/sys' for sysfs root. Jan 13 09:18:39.048 INFO vector::app: Loading configs. path=[("/etc/vector/vector.toml", Some(Toml))] Jan 13 09:18:39.051 ERROR vector::cli: Configuration error. error=expected an equals, found a right brace at line 59 column 33
Generated vector.toml file:
# Logs from containers (including openshift containers) [sources.raw_container_logs] type = "kubernetes_logs" auto_partial_merge = true exclude_paths_glob_patterns = ["/var/log/pods/openshift-logging_collector-*/*/*.log", "/var/log/pods/openshift-logging_elasticsearch-*/*/*.log", "/var/log/pods/openshift-logging_kibana-*/*/*.log"][transforms.container_logs] type = "remap" inputs = ["raw_container_logs"] source = ''' level = "unknown" if match(.message,r'(Warning|WARN|W[0-9]+|level=warn|Value:warn|"level":"warn")'){ level = "warn" } else if match(.message, r'Info|INFO|I[0-9]+|level=info|Value:info|"level":"info"'){ level = "info" } else if match(.message, r'Error|ERROR|E[0-9]+|level=error|Value:error|"level":"error"'){ level = "error" } else if match(.message, r'Debug|DEBUG|D[0-9]+|level=debug|Value:debug|"level":"debug"'){ level = "debug" } .level = level .pipeline_metadata.collector.name = "vector" .pipeline_metadata.collector.version = "0.14.1" ip4, err = get_env_var("NODE_IPV4") .pipeline_metadata.collector.ipaddr4 = ip4 received, err = format_timestamp(now(),"%+") .pipeline_metadata.collector.received_at = received .pipeline_metadata.collector.error = err ''' [transforms.route_container_logs] type = "route" inputs = ["container_logs"] route.app = '!(starts_with!(.kubernetes.pod_namespace,"kube") && starts_with!(.kubernetes.pod_namespace,"openshift") && .kubernetes.pod_namespace == "default")' # Rename log stream to "application" [transforms.application] type = "remap" inputs = ["route_container_logs.app"] source = """ .log_type = "application" """ [transforms.to-loki] type = "remap" inputs = ["application"] source = """ . """[sinks.loki_server] type = "loki" inputs = ["to-loki"] endpoint = "http://loki-ingester-http-lokistack-dev.lokicluster.svc:3100" tenant_id = {kubernetes.pod_name}[sinks.loki_server.encoding] codec = "json"[sinks.loki_server.labels] kubernetes_container_name = "{{kubernetes.container_name}}" kubernetes_host = "${NODE_NAME}" kubernetes_namespace_name = "{{kubernetes.namespace_name}}" kubernetes_pod_name = "{{kubernetes.pod_name}}" log_type = "{{log_type}}"