Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-2607

[release-5.4] $labels.instance is empty in the message when firing FluentdNodeDown alert

    XMLWordPrintable

Details

    • False
    • None
    • False
    • NEW
    • VERIFIED
    • Before this update, the `FluentdNodeDown` alert could not provide instance labels in the message section appropriately. This update resolves the issue by fixing the alert rule to provide instance labels in cases of partial instance failures.
    • Logging (Core) - Sprint 218, Logging (Core) - Sprint 219

    Description

      Description of problem:

      When firing FluentdNodeDown alert, the alert message is missing "$labels.instance" as follows.
      As a result, it does not honor "$labels.instance" label in the alert message. Check the image capture file together please.

      Displayed messages:
      ~~~
      Prometheus could not scrape fluentd for more than 10m.
      ~~~

      Alert rule definition:
      ~~~

      name: logging_fluentd.alerts
      rules:
      alert: FluentdNodeDown
      annotations:
      message: Prometheus could not scrape fluentd {{ $labels.instance }} for more <--- HERE
      than 10m.
      summary: Fluentd cannot be scraped
      expr: absent(up

      {job="collector",namespace="openshift-logging"}

      == 1)
      for: 10m
      labels:
      namespace: openshift-logging
      service: fluentd
      severity: critical
      ~~~
      Is it expected result ? How to display the "$labels.instance" label in the alert message ?

      Version-Release number of selected component (if applicable):

      $ oc version
      Client Version: 4.9.23
      Server Version: 4.9.23
      Kubernetes Version: v1.22.3+b93fd35

      $ oc get csv
      NAME DISPLAY VERSION REPLACES PHASE
      cluster-logging.5.4.0-138 Red Hat OpenShift Logging 5.4.0-138 cluster-logging.5.3.5-20 Succeeded
      elasticsearch-operator.5.4.0-152 OpenShift Elasticsearch Operator 5.4.0-152 elasticsearch-operator.5.3.5-20 Succeeded

      How reproducible:

      You can always reproduce this when firing the "FluentdNodeDown".

      Steps to Reproduce:
      1.
      2.
      3.

      Actual results:

      Missing the "$labels.instance" value in the alert messages, even though the label is configured in the alert rules.

      Expected results:

      The "$labels.instance" value should be displayed as the alert rule is defined.

      Additional info:

      Attachments

        Activity

          People

            rhn-support-dapark Daein Park
            bysnupy Daein Park (Inactive)
            Anping Li Anping Li
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: