Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-7372

Re using loki recording rule in loki alerting rule causes field validation on "expr" to fail

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Not a Bug
    • Icon: Undefined Undefined
    • None
    • None
    • Log Storage
    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • False
    • NEW
    • NEW
    • Moderate

      Create loki recording rule: 

      apiVersion: loki.grafana.com/v1
      kind: RecordingRule
      metadata:
        name: test-recordingrule
        namespace: ns-testapp-logalert
        labels: 
          openshift.io/log-alerting: 'true'
      spec:
        groups:
          - interval: 1m
            name: Test-custom-recordingrule
            rules:
              - expr: |
                  sum(rate(\{kubernetes_namespace_name="ns-testapp-logalert", kubernetes_pod_name=~"testapp-logalerting.*"} |= "error" [1m]))
                record: 'nstestapp:error:rate1m'
        tenantID: application
      

      Attempt to use the recorded name in an alertingRule

      apiVersion: loki.grafana.com/v1
      kind: AlertingRule
      metadata:
        name: test-logging-alert
        namespace: ns-testapp-logalert
        labels: 
          openshift.io/log-alerting: 'true'
      spec:
        groups:
          - interval: 1m
            name: Testloggingalert
            rules:
              - alert: TestAppHighErrorRate
                annotations:
                  description: testing1,2
                  summary: testing1,2
                expr: >
                   nstestapp:error:rate1m > 0.01 
                for: 1m
                labels:
                  severity: critical
        tenantID: application
      

      Even if I add `kubernetes_namespace_name` which I would not expect to have to do, then the validaton still fails:

      apiVersion: loki.grafana.com/v1
      kind: AlertingRule
      metadata:
        name: test-logging-alert
        namespace: ns-testapp-logalert
        labels: 
          openshift.io/log-alerting: 'true'
      spec:
        groups:
          - interval: 1m
            name: Testloggingalert
            rules:
              - alert: TestAppHighErrorRate
                annotations:
                  description: testing1,2
                  summary: testing1,2
                expr: >
                   nstestapp:error:rate1m{kubernetes_namespace_name="ns-testapp-logalert"} > 0.01 
                for: 1m
                labels:
                  severity: critical
        tenantID: application
      

      I dont see a test here for using a recording rule in an alert: and all the test seem to require the label `kubernetes_namespace_name`
      but I am already using this in the recording rule, so would not expect to have to use it again.

      https://github.com/grafana/loki/blob/main/operator/internal/validation/openshift/recordingrule_test.go
      https://github.com/grafana/loki/blob/main/operator/internal/validation/openshift/recordingrule.go

              Unassigned Unassigned
              rhn-support-nigsmith Nigel Smith
              Anping Li Anping Li
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: