Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-5588

Fluentd to Vector JSON format change investigation

XMLWordPrintable

    • False
    • None
    • False
    • NEW
    • NEW
    • Bug Fix

      Proposed title of this feature request

      cx noticed log JSON change after migrating fluentd to vector.
      
      For instance:
      ・istiod.log
      before:
      ip-10-249-81-55.ap-northeast-1.compute.internal istiod-basic-7cd97fbd-89nvb 2024-05-16T05:24:35.873035Z info ads RDS: PUSH request for node:httpd-test-64f98c577d-qnlkv.hitachi-test resources:16 size:24.7kB cached:0/16
      after:
      istiod-basic-7cd97fbd-89nvb 2024-05-16T05:24:35.873035Z info ads RDS: PUSH request for node:httpd-test-64f98c577d-qnlkv.hitachi-test resources:16 size:24.7kB cached:0/16
      
      ⇒ip-10-249-81-55.ap-northeast-1.compute.internal is gone.
      

      What is the nature and description of the request?

      Workaround:
      made change to rsyslog.conf(cx's output server) as follows:
      property(name="$!kubernetes!host")
      ↓
      property(name="$!hostname")
      
      Reason for the WA can see host format is different:
      Fluentd JSON:
      {
        "_index": "app-000001",
        "_type": "_doc",
        "_id": "NTk2NWI1ODktZDk1My00MjcyLWE4ZmQtNThjYWRiZDVlYTAw",
        "_version": 1,
        "_score": null,
        "_source": {
          "kubernetes": {
            "container_image_id": "registry.redhat.io/openshift-service-mesh/pilot-rhel8@sha256:b451f504cb2078f363a4b184e4223289908ddf499f68bdee99f65350035276a6",
            "container_name": "discovery",
            "namespace_id": "1dcdcbf8-9f58-456a-9a6f-21e00fdbd669",
            "flat_labels": [
              "app=istiod",
              "istio=istiod",
              "istio_io_rev=basic",
              "maistra-control-plane=istio-system",
              "pod-template-hash=c754cb9cf",
              "sidecar_istio_io_inject=false"
            ],
            "pod_ip": "10.131.2.31",
            "host": "hollytest-wbxlb-worker-eastus1-j8r6x",
            "master_url": "https://kubernetes.default.svc",
            "pod_id": "f051a874-95f1-4817-9bef-69ed48a46827",
            "namespace_labels": {
              "pod-security_kubernetes_io_audit": "restricted",
              "maistra_io_member-of": "istio-system",
              "pod-security_kubernetes_io_warn-version": "v1.24",
              "pod-security_kubernetes_io_warn": "restricted",
              "kubernetes_io_metadata_name": "istio-system",
              "kiali_io_member-of": "istio-system",
              "pod-security_kubernetes_io_audit-version": "v1.24"
            },
            "container_image": "registry.redhat.io/openshift-service-mesh/pilot-rhel8@sha256:f12207461e6613649d6c62eea7bdf00829715249068796d59ba2fe99c19948ae",
            "namespace_name": "istio-system",
            "pod_name": "istiod-basic-c754cb9cf-xbwqr"
          },
          "viaq_msg_id": "NTk2NWI1ODktZDk1My00MjcyLWE4ZmQtNThjYWRiZDVlYTAw",
          "level": "info",
          "openshift": {
            "sequence": 585,
            "cluster_id": "938c0ea1-838f-420c-a1ff-2352058b517b"
          },
          "message": "2024-05-17T08:09:03.573347Z\tinfo\tads\tLDS: PUSH for node:istio-ingressgateway-69f8699479-bhshr.istio-system resources:1 size:2.5kB",  
          "docker": {
            "container_id": "bbb58ae23d1226db7f13d5193a0c5ec4ac6f10d67fee321a8b41361f55498c45"
          },
          "hostname": "hollytest-wbxlb-worker-eastus1-j8r6x",
          "log_type": "application",
          "@timestamp": "2024-05-17T08:09:03.573383221+00:00",
          "pipeline_metadata": {
            "collector": {
              "received_at": "2024-05-17T08:09:03.579057+00:00",
              "name": "fluentd",
              "inputname": "fluent-plugin-systemd",
              "version": "1.16.2 1.6.0",
              "ipaddr4": "10.0.4.136"
            }
          }
        },
        "fields": {
          "@timestamp": [
            "2024-05-17T08:09:03.573Z"
          ],
          "pipeline_metadata.collector.received_at": [
            "2024-05-17T08:09:03.579Z"
          ]
        },
        "highlight": {
          "message": [
            "2024-05-17T08:09:03.573347Z\tinfo\tads\tLDS: @kibana-highlighted-field@PUSH@/kibana-highlighted-field@ for node:istio-ingressgateway-69f8699479-bhshr.istio-system resources:1 size:2.5kB"
          ],
          "kubernetes.namespace_name": [
            "@kibana-highlighted-field@istio-system@/kibana-highlighted-field@"
          ]
        },
        "sort": [
          1715933343573
        ]
      }
      
      ------------------------------------------
      
      Vector JSON:
      
      {
        "_index": "app-000001",
        "_type": "_doc",
        "_id": "YzRkZWI0N2UtYmUxMS00YWI4LTg2MzAtYmY3MWVkZTc3ZmI1",
        "_version": 1,
        "_score": null,
        "_source": {
          "kubernetes": {
            "pod_ip": "10.131.1.143",
            "annotations": {
              "seccomp.security.alpha.kubernetes.io/pod": "runtime/default",
              "prometheus.io/port": "15014",
              "sidecar.istio.io/inject": "false",
              "ambient.istio.io/redirection": "disabled",
              "prometheus.io/scrape": "true",
              "openshift.io/scc": "restricted-v2",
              "k8s.ovn.org/pod-networks": "{\"default\":{\"ip_addresses\":[\"10.131.1.143/23\"],\"mac_address\":\"0a:58:0a:83:01:8f\",\"gateway_ips\":[\"10.131.0.1\"],\"ip_address\":\"10.131.1.143/23\",\"gateway_ip\":\"10.131.0.1\"}}",
              "k8s.v1.cni.cncf.io/network-status": "[{\n    \"name\": \"ovn-kubernetes\",\n    \"interface\": \"eth0\",\n    \"ips\": [\n        \"10.131.1.143\"\n    ],\n    \"mac\": \"0a:58:0a:83:01:8f\",\n    \"default\": true,\n    \"dns\": {}\n}]"
            },
            "pod_owner": "ReplicaSet/istiod-basic-5794cf67bb",
            "namespace_labels": {
              "pod-security_kubernetes_io_audit": "restricted",
              "maistra_io_member-of": "istio-system",
              "pod-security_kubernetes_io_warn-version": "v1.24",
              "pod-security_kubernetes_io_warn": "restricted",
              "kubernetes_io_metadata_name": "istio-system",
              "kiali_io_member-of": "istio-system",
              "pod-security_kubernetes_io_audit-version": "v1.24"
            },
            "container_image": "registry.redhat.io/openshift-service-mesh/pilot-rhel8@sha256:f12207461e6613649d6c62eea7bdf00829715249068796d59ba2fe99c19948ae",
            "labels": {},
            "namespace_name": "istio-system",
            "pod_name": "istiod-basic-5794cf67bb-7kt4g",
            "container_name": "discovery",
            "flat_labels": [
              "istio=istiod",
              "pod-template-hash=5794cf67bb",
              "app=istiod",
              "istio_io_rev=basic",
              "maistra-control-plane=istio-system",
              "sidecar_istio_io_inject=false"
            ],
            "namespace_id": "1dcdcbf8-9f58-456a-9a6f-21e00fdbd669",
            "pod_id": "d2f7d700-cd5a-4ed5-848e-14d9825fe264",
            "container_id": "cri-o://6e4df9af8a5eaba793eee288a867abb15fcacc98dee530422f2bf63c5d615ef0"
          },
          "hostname": "hollytest-wbxlb-worker-eastus3-nh85b",
          "log_type": "application",
          "@timestamp": "2024-05-17T07:45:43.319246645Z",
          "level": "info",
          "openshift": {
            "sequence": 139860,
            "cluster_id": "938c0ea1-838f-420c-a1ff-2352058b517b"
          },
          "message": "2024-05-17T07:45:43.319208Z\tinfo\tads\tSDS: PUSH request for node:istio-ingressgateway-5bdb858658-dnnqh.istio-system resources:1 size:2.8kB cached:1/1"   
        },
        "fields": {
          "@timestamp": [
            "2024-05-17T07:45:43.319Z"
          ]
        },
        "highlight": {
          "kubernetes.namespace_name": [
            "@kibana-highlighted-field@istio-system@/kibana-highlighted-field@"
          ]
        },
        "sort": [
          1715931943319
        ]
      }
      
      

      Why does the customer need this? (List the business requirements)

      cx would like to make less change to their rsyslog conf as much as possible(keep their old setting with fluentd) and asked whether change from vector JSON can be made to let them use old setting with fluentd to work well with vector as well.

      List any affected packages or components.

      like for instance host:
      Fluentd: kubernetes:

      {host:}

      Vector: kubernetes:{} hostname:

              Unassigned Unassigned
              rhn-support-hqiao Holly Qiao
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: