Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-4429

[release-5.8] Infrastructure tenant alerts are firing under application namespace on dev-console

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Critical Critical
    • Logging 5.8.0
    • Logging 5.8.0
    • Log Console
    • None
    • False
    • Hide

      None

      Show
      None
    • False
    • NEW
    • VERIFIED
    • Release Note Not Required

      Description:

      Infrastructure tenant alerts are firing under application namespace on dev-console

      Rules are created under my-app (application) and openshift-logging (infrastructure) namespace. Infra rule cannot be seen firing under openshift-logging namespace but under my-app(application) under dev-console.

      Request URL: https://console-openshift-console.apps.kbharti-0807gcp.qe.gcp.devcluster.openshift.com/api/proxy/plugin/logging-view-plugin/backend/api/logs/v1/infrastructure/prometheus/api/v1/rules?kubernetes_namespace_name=my-app

      Response: http://pastebin.test.redhat.com/1106828 

      Steps to Reproduce:
      1) Forward logs to default logStore loki
      2) Create app and infra alerting rules 
      3) Verify alerts firing under dev-console under respective namespace.

      $ oc get csv -n openshift-logging
      NAME                     DISPLAY                     VERSION   REPLACES   PHASE
      cluster-logging.v5.8.0   Red Hat OpenShift Logging   5.8.0                Succeeded
      loki-operator.v5.8.0     Loki Operator               5.8.0                Succeeded

       

      $ oc extract cm/lokistack-dev-rules-0 
      openshift-logging-infra-workload-alerts-4b4201b3-f7b0-4917-b232-84610021e47e.yaml
      my-app-dev-workload-alerts-ae5fc6e3-72c7-40cf-bb0d-56c45ef2db3d.yaml
      [kbharti@desktop loki-alerts]$ cat openshift-logging-infra-workload-alerts-4b4201b3-f7b0-4917-b232-84610021e47e.yaml 
      groups:
      
      name: InfraAppAlert
        interval: 1m
        limit: 0
        rules:
        - alert: InfraLogVolumeIsHigh
          expr: |
            count_over_time({kubernetes_namespace_name="openshift-logging"}[2m]) > 0
          for: 5m
          annotations:
            description: openshift-logging namespace has high amount of logs.
            summary: project "openshift-logging" log volume is high.
          labels:
            devApp: "false"
            severity: info
            tenantId: infrastructure
      [kbharti@desktop loki-alerts]$ cat my-app-dev-workload-alerts-ae5fc6e3-72c7-40cf-bb0d-56c45ef2db3d.yaml 
      groups: name: devAppAlert
        interval: 1m
        limit: 0
        rules:
        - alert: DevAppLogVolumeIsHigh
          expr: |
            count_over_time({kubernetes_namespace_name="my-app"}[2m]) > 10
          for: 5m
          annotations:
            description: My application has high amount of logs.
            summary: project "my-app" log volume is high.
          labels:
            devApp: "true"
            kubernetes_namespace_name: my-app
            severity: info
            tenantId: application 
      $ oc logs logging-view-plugin-56cfc49957-gtqdt | head -5
      time="2023-08-07T11:18:54Z" level=info msg="enabled features: [\"alerts\" \"dev-console\" \"dev-alerts\"]\n" module=main
      time="2023-08-07T11:18:54Z" level=info msg="listening on https://:9443" module=server
      10.129.0.98 - - [07/Aug/2023:11:19:01 +0000] "GET /plugin-manifest.json HTTP/1.1" 200 2494
      10.129.0.98 - - [07/Aug/2023:11:19:16 +0000] "GET /plugin-manifest.json HTTP/1.1" 200 2494
      10.129.0.98 - - [07/Aug/2023:11:19:16 +0000] "GET /exposed-getUserAlertingRules-chunk-690f2d40ee1a1e71d576.min.js HTTP/1.1" 200 0
      ....

      How reproducible: Always

      Actual Result: Infra alert rule for openshift-logging project can be seen firing under app namespace.

      Expected Result: Alerting rules should fire under respective app namespaces.

        1. Screenshot 2023-08-07 at 5.20.15 PM.png
          458 kB
          Kabir Bharti
        2. Screenshot 2023-08-07 at 5.20.04 PM.png
          456 kB
          Kabir Bharti
        3. Screenshot 2023-10-16 at 5.05.55 PM.png
          394 kB
          Kabir Bharti

              gbernal@redhat.com Gabriel Bernal
              rhn-support-kbharti Kabir Bharti
              Kabir Bharti Kabir Bharti
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: