Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-6330

Prune filter does not prune OTEL attributes on log streams.

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Not a Bug
    • Icon: Blocker Blocker
    • None
    • Logging 6.1.0
    • Log Collection
    • False
    • None
    • False
    • NEW
    • OBSDA-849 - Enable Full In-Cluster OpenTelemetry Support In OpenShift Logging
    • NEW
    • Hide
      Disclaimer for release notes and OTEL doc section (reformat appropriately) see also LOG-6341:
      -----
      The tech preview OTEL support allows you to store and view logs in OTEL format.
      The forwarder still uses the Viaq model internally, and translates to OTEL before sending.

      You need to use Viaq labels instead of OTEL labels:
      - In filters in your CLF configuration. The filter still affects equivalent fields in the OTEL output.
      - In alerting rules, in particular for authorization purposes your filter must include `kubernetes_namespace_name` not `k8s_namespace_name`.

      FIXME: xref to relevant doc for filters & alerting rules
      Show
      Disclaimer for release notes and OTEL doc section (reformat appropriately) see also LOG-6341 : ----- The tech preview OTEL support allows you to store and view logs in OTEL format. The forwarder still uses the Viaq model internally, and translates to OTEL before sending. You need to use Viaq labels instead of OTEL labels: - In filters in your CLF configuration. The filter still affects equivalent fields in the OTEL output. - In alerting rules, in particular for authorization purposes your filter must include `kubernetes_namespace_name` not `k8s_namespace_name`. FIXME: xref to relevant doc for filters & alerting rules
    • Known Issue
    • Proposed

      Description: Prune filter does not prune OTEL attributes on log streams.

      CLF:

      spec:
        filters:
          - name: prune-logs
            prune:
              in:
                - .k8s.container.name
            type: prune
          - name: my-custom-labels
            openshiftLabels:
              custom-qe-labels-added: 'true'
            type: openshiftLabels
        managementState: Managed
        outputs:
          - lokiStack:
              authentication:
                token:
                  from: serviceAccount
              dataModel: Otel
              target:
                name: lokistack-dev
                namespace: openshift-logging
              tuning:
                compression: none
            name: lokistack-otlp
            tls:
              ca:
                configMapName: openshift-service-ca.crt
                key: service-ca.crt
            type: lokiStack
        pipelines:
          - filterRefs:
              - my-custom-labels
              - prune-logs
            inputRefs:
              - application
              - infrastructure
              - audit
            name: my-pipeline
            outputRefs:
              - lokistack-otlp
        serviceAccount:
          name: loki-otlp

      Here, we expect the OTEL attribute k8s.container.name to be pruned on log streams but it still continues to exist on logs.

      Logs:

      $ logcli --tls-skip-verify --bearer-token="$(oc whoami -t)" --addr "https://lokistack-dev-openshift-logging.apps.kbharti1029x.qe.devcluster.openshift.com/api/logs/v1/application" query '{k8s_namespace_name="my-app", k8s_container_name="centos-logtest"}' | tail -1
      
      2024/10/29 14:28:07 https://lokistack-dev-openshift-logging.apps.kbharti1029x.qe.devcluster.openshift.com/api/logs/v1/application/loki/api/v1/query_range?direction=BACKWARD&end=1730226487909182000&limit=30&query=%7Bk8s_namespace_name%3D%22my-app%22%2C+k8s_container_name%3D%22centos-logtest%22%7D&start=1730222887909182000
      2024/10/29 14:28:08 Common labels: {detected_level="debug", k8s_container_name="centos-logtest", k8s_namespace_name="my-app", k8s_node_name="ip-10-0-3-101.us-east-2.compute.internal", k8s_pod_label_fix_env_qe_ispresent="true", k8s_pod_name="centos-logtest-chx2t", k8s_pod_uid="2635a7ff-476c-4d3d-94a3-fb88b52f2652", kubernetes_container_name="centos-logtest", kubernetes_host="ip-10-0-3-101.us-east-2.compute.internal", kubernetes_namespace_name="my-app", kubernetes_pod_name="centos-logtest-chx2t", log_iostream="stdout", log_source="container", log_type="application", openshift_cluster_id="016174f6-2bb9-4cf0-aa7d-33531936f240", openshift_cluster_uid="016174f6-2bb9-4cf0-aa7d-33531936f240", openshift_label_custom_qe_labels_added="true", openshift_log_source="container", openshift_log_type="application", severity_text="default"}
      2024-10-29T14:24:49-04:00 {observed_timestamp="1730226289705581596"} {"message": "MERGE_JSON_LOG=true", "level": "debug","Layer1": "layer1 0", "layer2":
      {"name":"Layer2 1", "tips":"Decide by PRESERVE_JSON_LOG"}
      , "StringNumber":"10", "Number": 10,"foo.bar":"Dot Item","{foobar}":"Brace Item","[foobar]":"Bracket Item", "foo:bar":"Colon Item","foo bar":"Space Item" }
      

       

      vector.toml

      ....
      [transforms.pipeline_my_pipeline_1_prune_logs_3]
      type = "remap"
      inputs = ["pipeline_my_pipeline_1_my_custom_labels_2"]
      source = '''
        in = [["k8s","container","name"]]
        # Remove keys from in list
        for_each(in) -> |_index, val|
      {       . = remove!(., val)   }
      '''
      ...
      

       

      Version: Logging 6.1

      Steps to reproduce:
      1) Deploy CLO and LO v6.1
      2) Create a CLF with a prune filter on one of the labels.
      3) Observe that the attribute is pruned.

      Actual Result: Logs still contain the pruned field

      Expected Result: Logs should not contain the attribute which is being pruned on filter.

      Additional Info: Using the viaq attribute 'kubernetes.pod_name', we can see the field being pruned.

              Unassigned Unassigned
              rhn-support-kbharti Kabir Bharti
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated:
                Resolved: