Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-6341

Unable to create AlertingRule with OTEL namespace identifier k8s_namespace_name

XMLWordPrintable

    • False
    • None
    • False
    • NEW
    • NEW
    • Hide
      Disclaimer for release notes and OTEL doc section (reformat appropriately), see also LOG-6330:
      -----
      The tech preview OTEL support allows you to store and view logs in OTEL format.
      The forwarder still uses the Viaq model internally, and translates to OTEL before sending.

      You need to use Viaq labels instead of OTEL labels:
      - In filters in your CLF configuration. The filter still affects equivalent fields in the OTEL output.
      - In alerting rules, in particular for authorization purposes your filter must include `kubernetes_namespace_name` not `k8s_namespace_name`.

      FIXME: xref to relevant doc for filters & alerting rules
      Show
      Disclaimer for release notes and OTEL doc section (reformat appropriately), see also LOG-6330 : ----- The tech preview OTEL support allows you to store and view logs in OTEL format. The forwarder still uses the Viaq model internally, and translates to OTEL before sending. You need to use Viaq labels instead of OTEL labels: - In filters in your CLF configuration. The filter still affects equivalent fields in the OTEL output. - In alerting rules, in particular for authorization purposes your filter must include `kubernetes_namespace_name` not `k8s_namespace_name`. FIXME: xref to relevant doc for filters & alerting rules
    • Known Issue
    • Proposed

      Description: 
      k8s.namespace.name is a required stream label for OTEL datamodel and is same as 'kubernetes.namespace_name' viaq attribute which is used under logQL expression for Alerts. Currently, creating an alerting rule with identifier k8s.namespace.name results in error.

      Error:

      Error "Invalid value: "count_over_time({k8s_namespace_name=\"my-app\"}[2m]) > 10\n": rule needs to have a matcher for the namespace" for field "spec.groups[0].rules[0].expr".

      Version: Loki Operator v6.1

      Steps to reproduce:
      a) Deploy CLO and LO v6.1
      b) Create LokiStack and corresponding Alertingrule with OTEL semantic convention namespace identifier.

      Expected Result: AlertingRule is created successfully and ruler is running succesfully after restart.

      Actual Result: AlertingRule creation errors out.

      AlertingRule:

      apiVersion: loki.grafana.com/v1
kind: AlertingRule
metadata:
  labels:
    openshift.io/cluster-monitoring: 'true'
  name: my-app-alert
  namespace: my-app
spec:
  groups:
    - interval: 1m
      name: MyApplication
      rules:
        - alert: MyAppLogVolumeIsHigh
          annotations:
            description: My app has high amount of logs.
            summary: project "my-app" log volume is high.
          expr: >
            count_over_time({k8s_namespace_name="my-app"}[2m])
            > 10
          for: 5m
          labels:
            severity: info
            project: my-app
  tenantID: application

              Unassigned Unassigned
              rhn-support-kbharti Kabir Bharti
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated: