Details
-
Bug
-
Resolution: Done
-
Normal
-
Logging 5.4.0
-
None
-
False
-
False
-
NEW
-
OBSDA-108 - Distribute an alternate Vector Log Collector
-
VERIFIED
-
Logging (Core) - Sprint 214
Description
The Vector config for Loki labels have wrong values for kubernetes_host and kubernetes_namespace_name. The kubernetes_namespace_name = "kubernetes.namespace_name" while it should be kubernetes_namespace_name = "pod_namespace". Following is the Vector config that is generated by CLF.
[sinks.loki_server.labels] kubernetes_container_name = "{{kubernetes.container_name}}" kubernetes_host = "${NODE_NAME}" kubernetes_namespace_name = "{{kubernetes.namespace_name}}" kubernetes_pod_name = "{{kubernetes.pod_name}}" log_type = "{{log_type}}"
Log shipped to Loki:
"stream": { "kubernetes_host": "", "kubernetes_pod_name": "centos-logtest-wbwmm", "log_type": "application", "kubernetes_container_name": "centos-logtest" }, "values": [ [ "1644919864220616906", "{\"file\":\"/var/log/pods/test_centos-logtest-wbwmm_c8b9a7fc-4353-4169-8737-e6aded653812/centos-logtest/0.log\",\"kubernetes.container_id\":\"cri-o://613bcdba11e5c3dbc1e030f2671f812155538a3be0affe8dbeee749f42f08842\",\"kubernetes.container_image\":\"quay.io/openshifttest/ocp-logtest@sha256:16232868ba1143721b786dbabb3f7384645acb663fadb4af48e9ea1228a67635\",\"kubernetes.container_name\":\"centos-logtest\",\"kubernetes.pod_ip\":\"10.129.2.65\",\"kubernetes.pod_ips[0]\":\"10.129.2.65\",\"kubernetes.pod_labels.run\":\"centos-logtest\",\"kubernetes.pod_labels.test\":\"centos-logtest\",\"kubernetes.pod_name\":\"centos-logtest-wbwmm\",\"kubernetes.pod_namespace\":\"test\",\"kubernetes.pod_node_name\":\"ip-10-0-219-209.us-east-2.compute.internal\",\"kubernetes.pod_uid\":\"c8b9a7fc-4353-4169-8737-e6aded653812\",\"level\":\"unknown\",\"log_type\":\"application\",\"message\":\"{\\\"message\\\": \\\"MERGE_JSON_LOG=true\\\", \\\"level\\\": \\\"debug\\\",\\\"Layer1\\\": \\\"layer1 0\\\", \\\"layer2\\\": {\\\"name\\\":\\\"Layer2 1\\\", \\\"tips\\\":\\\"Decide by PRESERVE_JSON_LOG\\\"}, \\\"StringNumber\\\":\\\"10\\\", \\\"Number\\\": 10,\\\"foo.bar\\\":\\\"Dot Item\\\",\\\"{foobar}\\\":\\\"Brace Item\\\",\\\"[foobar]\\\":\\\"Bracket Item\\\", \\\"foo:bar\\\":\\\"Colon Item\\\",\\\"foo bar\\\":\\\"Space Item\\\" }\",\"pipeline_metadata.collector.error\":null,\"pipeline_metadata.collector.ipaddr4\":\"10.0.219.209\",\"pipeline_metadata.collector.name\":\"vector\",\"pipeline_metadata.collector.received_at\":\"2022-02-15T10:11:04.387961378+00:00\",\"pipeline_metadata.collector.version\":\"0.14.1\",\"source_type\":\"kubernetes_logs\",\"stream\":\"stdout\"}" ]
Steps to reproduce the issue.
*Install a Loki server.
*Forward logs to the Loki server with Vector as collector.
apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
name: instance
namespace: openshift-logging
spec:
outputs:
outputs:
- name: loki-server
type: loki
url: http://loki-server.loki-aosqe.svc:3100
pipelines:
- name: to-loki
inputRefs:
- application
- audit
- infrastructure
outputRefs:
- loki-server
*Check the generated Vector config and the log record in Loki.
Vector Loki labels config:
[sinks.loki_server.labels] kubernetes_container_name = "{{kubernetes.container_name}}" kubernetes_host = "${NODE_NAME}" kubernetes_namespace_name = "{{kubernetes.namespace_name}}" kubernetes_pod_name = "{{kubernetes.pod_name}}" log_type = "{{log_type}}"
Logs shipped to loki:
"stream": { "kubernetes_host": "", "kubernetes_pod_name": "centos-logtest-wbwmm", "log_type": "application", "kubernetes_container_name": "centos-logtest" }, "values": [ [ "1644919864220616906", "{\"file\":\"/var/log/pods/test_centos-logtest-wbwmm_c8b9a7fc-4353-4169-8737-e6aded653812/centos-logtest/0.log\",\"kubernetes.container_id\":\"cri-o://613bcdba11e5c3dbc1e030f2671f812155538a3be0affe8dbeee749f42f08842\",\"kubernetes.container_image\":\"quay.io/openshifttest/ocp-logtest@sha256:16232868ba1143721b786dbabb3f7384645acb663fadb4af48e9ea1228a67635\",\"kubernetes.container_name\":\"centos-logtest\",\"kubernetes.pod_ip\":\"10.129.2.65\",\"kubernetes.pod_ips[0]\":\"10.129.2.65\",\"kubernetes.pod_labels.run\":\"centos-logtest\",\"kubernetes.pod_labels.test\":\"centos-logtest\",\"kubernetes.pod_name\":\"centos-logtest-wbwmm\",\"kubernetes.pod_namespace\":\"test\",\"kubernetes.pod_node_name\":\"ip-10-0-219-209.us-east-2.compute.internal\",\"kubernetes.pod_uid\":\"c8b9a7fc-4353-4169-8737-e6aded653812\",\"level\":\"unknown\",\"log_type\":\"application\",\"message\":\"{\\\"message\\\": \\\"MERGE_JSON_LOG=true\\\", \\\"level\\\": \\\"debug\\\",\\\"Layer1\\\": \\\"layer1 0\\\", \\\"layer2\\\": {\\\"name\\\":\\\"Layer2 1\\\", \\\"tips\\\":\\\"Decide by PRESERVE_JSON_LOG\\\"}, \\\"StringNumber\\\":\\\"10\\\", \\\"Number\\\": 10,\\\"foo.bar\\\":\\\"Dot Item\\\",\\\"{foobar}\\\":\\\"Brace Item\\\",\\\"[foobar]\\\":\\\"Bracket Item\\\", \\\"foo:bar\\\":\\\"Colon Item\\\",\\\"foo bar\\\":\\\"Space Item\\\" }\",\"pipeline_metadata.collector.error\":null,\"pipeline_metadata.collector.ipaddr4\":\"10.0.219.209\",\"pipeline_metadata.collector.name\":\"vector\",\"pipeline_metadata.collector.received_at\":\"2022-02-15T10:11:04.387961378+00:00\",\"pipeline_metadata.collector.version\":\"0.14.1\",\"source_type\":\"kubernetes_logs\",\"stream\":\"stdout\"}" ]
The log record that is generated.
"kubernetes": { "pod_namespace": "test", "pod_ips": [ "10.129.2.65" ], "container_name": "centos-logtest", "flat_labels": [ "run=centos-logtest", "test=centos-logtest" ], "pod_uid": "c8b9a7fc-4353-4169-8737-e6aded653812", "pod_ip": "10.129.2.65", "container_id": "cri-o://613bcdba11e5c3dbc1e030f2671f812155538a3be0affe8dbeee749f42f08842", "container_image": "quay.io/openshifttest/ocp-logtest@sha256:16232868ba1143721b786dbabb3f7384645acb663fadb4af48e9ea1228a67635", "pod_name": "centos-logtest-wbwmm", "pod_node_name": "ip-10-0-219-209.us-east-2.compute.internal" },
Attachments
Issue Links
- links to
- mentioned on
(3 mentioned on)
