Uploaded image for project: 'Observability and Data Analysis Program'
  1. Observability and Data Analysis Program
  2. OBSDA-1236

Support for enabling the structured_metadata option in the Vector

XMLWordPrintable

    • Icon: Feature Feature
    • Resolution: Unresolved
    • Icon: Undefined Undefined
    • None
    • None
    • PM Logging
    • False
    • Hide

      None

      Show
      None
    • False
    • Not Selected
    • 0

      Proposed title of this feature request

      Support for enabling the structured_metadata option in the Vector

      What is the nature and description of the request?
      Option to enable structured_metadata = true for vector (sink=loki) via CLF . Below option is there in Upstream https://vector.dev/docs/reference/configuration/sinks/loki/#structured_metadata

      From loki end for version 6.3 and schema v13 below is enabled:
      % oc get cm logging-loki-config -oyaml | grep -i structured allow_structured_metadata: true

      Loki needs allow_structured_metadata: true  to accept structured metadata. (which we have in 6.3)
      But Vector needs structured_metadata = true to actually send it.
      Why does the customer need this? (List the business requirements)

      The need for structured metadata is as a replacement for sending labels for kubernetes-specific data (e.g. openshift pod name, resource owner, node etc.). Although necessary for effective querying, in loki a high number of changing labels is regarded as an anti-pattern and can lead to excessive object storage usage, degradation in loki's performance and very slow queries, as explained here: https://grafana.com/docs/loki/latest/get-started/labels/structured-metadata/
      In order to build useful dashboards and alerts with openshift logs while avoiding the slowness and costs associated with a large number of dynamic labels (especially for large clusters, with hundreds of namespaces and thousands of short-lived pods) structured metadata is required.

      List any affected packages or components.

      Openshift Logging, Vector

              rh-ee-erevans Eric Evans
              rhn-support-anisal Apurva Nisal
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated: