Uploaded image for project: 'Distributed Tracing'
  1. Distributed Tracing
  2. TRACING-4945

Use OTLP/http for sending logs to the LokiStack 6.1

XMLWordPrintable

    • Icon: Epic Epic
    • Resolution: Unresolved
    • Icon: Undefined Undefined
    • rhosdt-3.5
    • None
    • None
    • None
    • Use OTLP/http for sending logs to the LokiStack 6.1
    • 1
    • False
    • None
    • False
    • Not Selected
    • To Do

      The logging OTLP model https://github.com/rhobs/observability-data-model/blob/main/cluster-logging.md 

      LokiStack OTLP docs https://loki-operator.dev/docs/open-telemetry.md/ 

       

      Acceptance criteria:

       

       

      apiVersion: rbac.authorization.k8s.io/v1
      kind: ClusterRole
      metadata:
        name: otel-collector-logs-writer
      rules:
       - apiGroups:
         - loki.grafana.com
         resourceNames:
         - logs
         resources:
         - application
         verbs:
         - create
       - apiGroups: [""]
         resources: ["pods", "namespaces", "nodes"]
         verbs: ["get", "watch", "list"]
       - apiGroups: ["apps"]
         resources: ["replicasets"]
         verbs: ["get", "list", "watch"]
      ---
      apiVersion: rbac.authorization.k8s.io/v1
      kind: ClusterRoleBinding
      metadata:
        name: otel-collector-logs-writer
      roleRef:
        apiGroup: rbac.authorization.k8s.io
        kind: ClusterRole
        name: otel-collector-logs-writer
      subjects:
        - kind: ServiceAccount
          name: otel-collector
          namespace: openshift-logging
      ---
      apiVersion: opentelemetry.io/v1beta1
      kind: OpenTelemetryCollector
      metadata:
        name: otel
        namespace: openshift-logging
      spec:
        config:
          extensions:
            bearertokenauth:
              filename: "/var/run/secrets/kubernetes.io/serviceaccount/token"
          receivers:
            otlp:
              protocols:
                grpc:
                  endpoint: 0.0.0.0:4317
                http:
                  endpoint: 0.0.0.0:4318
          processors:
            k8sattributes: {}
            resource:
              attributes:
                - key:  kubernetes.namespace_name
                  from_attribute: k8s.namespace.name
                  action: upsert
                - key:  kubernetes.pod_name
                  from_attribute: k8s.pod.name
                  action: upsert
                - key: kubernetes.container_name
                  from_attribute: k8s.container.name
                  action: upsert
                - key: log_type
                  value: application
                  action: upsert
          exporters:
            otlphttp:
              endpoint: https://logging-loki-gateway-http.openshift-logging.svc.cluster.local:8080/api/logs/v1/application/otlp
              encoding: json
              tls:
                ca_file: "/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt"
              auth:
                authenticator: bearertokenauth
          service:
            extensions: [bearertokenauth]
            pipelines:
              logs:
                receivers: [otlp]
                processors: [k8sattributes, resource]
                exporters: [debug,otlphttp] 

       

      LokiStack creates a following config for loki

        otlp_config:
          resource_attributes:
            attributes_config:
            - action: index_label
              attributes:
              - k8s.container.name
              - k8s.cronjob.name
              - k8s.daemonset.name
              - k8s.deployment.name
              - k8s.job.name
              - k8s.namespace.name
              - k8s.node.name
              - k8s.pod.name
              - k8s.statefulset.name
              - kubernetes.container_name
              - kubernetes.host
              - kubernetes.namespace_name
              - kubernetes.pod_name
              - log_source
              - log_type
              - openshift.cluster.uid
              - openshift.log.source
              - openshift.log.type
              - service.name
            - action: structured_metadata
              attributes:
              - k8s.node.uid
              - k8s.pod.uid
              - k8s.replicaset.name
              - process.command_line
              - process.executable.name
              - process.executable.path
              - process.pid
            - action: structured_metadata
              regex: k8s\.pod\.labels\..+
            - action: structured_metadata
              regex: openshift\.labels\..+
          log_attributes:
          - action: structured_metadata
            attributes:
            - k8s.event.level
            - k8s.event.object_ref.api.group
            - k8s.event.object_ref.api.version
            - k8s.event.object_ref.name
            - k8s.event.object_ref.resource
            - k8s.event.request.uri
            - k8s.event.response.code
            - k8s.event.stage
            - k8s.event.user_agent
            - k8s.user.groups
            - k8s.user.username
            - log.iostream
          - action: structured_metadata
            regex: k8s\.event\.annotations\..+
          - action: structured_metadata
            regex: systemd\.t\..+
          - action: structured_metadata
            regex: systemd\.u\..+
        allow_structured_metadata: true 

              ploffay@redhat.com Pavol Loffay
              ploffay@redhat.com Pavol Loffay
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated: