Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-7012

[Logging 6.2] Parsing errors under Loki distributor for audit tenant when using Otel DataModel

XMLWordPrintable

    • Future Sustainability
    • False
    • Hide

      None

      Show
      None
    • False
    • NEW
    • VERIFIED
    • Before this fix, logs lacking a responseStatus.code field resulted in parsing errors within the Loki distributor when using the otel data model. With this fix those logs are now parsed correctly.
    • Bug Fix
    • Log Collection - Sprint 269, Log Collection - Sprint 270

      Description of problem:

      When forwarding Logs to Loki using Otel dataModel, distributor shows some parsing errors on audit log attributes. The attribute k8s.audit.event.response.code can be seen under audit logs though.

      Image: loki-rhel9-operator/images/v6.2.1-4

      Error:

      level=error ts=2025-04-11T18:55:10.95667372Z caller=manager.go:50 component=distributor path=write msg="write operation failed" details="couldn't parse push request: ReadInt64: strconv.ParseInt: parsing \"\": invalid syntax, error found in #10 byte of ...|tValue\":\"\"}},{\"key\":|..., bigger context ...|.audit.event.response.code\",\"value\":{\"intValue\":\"\"}},{\"key\":\"k8s.audit.event.user_agent\",\"value\":{\"s|..." org_id=audit

      Steps to reproduce:
      a) Deploy CLO and Loki Operator v6.2
      b) Forward logs to Loki using Otel dataModel
      c) After 10-15 minutes, observe Loki distributor logs.

      Version: Logging 6.2.1

      Actual Result: parsing error under distributor component for k8s.audit.event.response.code attribute

      Expected Result: No error should be seen under Loki components 

      Additional Info: Distributor logs attached.

      CLF:

      $ oc get clusterlogforwarders.observability.openshift.io logging-clf -o yaml | yq -e .spec
collector:
  resources:
    limits:
      cpu: 100m
      memory: 512Mi
    requests:
      cpu: 100m
      memory: 256Mi
managementState: Managed
outputs:
  - lokiStack:
      authentication:
        token:
          from: serviceAccount
      dataModel: Otel
      target:
        name: logging-loki
        namespace: openshift-logging
    name: output-to-lokistack
    tls:
      ca:
        configMapName: openshift-service-ca.crt
        key: service-ca.crt
    type: lokiStack
pipelines:
  - inputRefs:
      - audit
    name: logs-to-loki
    outputRefs:
      - output-to-lokistack
serviceAccount:
  name: loki-otlp

        1. Distributor_logs.txt
          66 kB

              rh-ee-calee Calvin Lee
              rhn-support-kbharti Kabir Bharti
              Kabir Bharti Kabir Bharti
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: