Description of problem:

The parsed field used as label in AlertingRule doesn't appear when the alert is triggered.

  - name: 404NotFound
    rules:
    - alert: TooMany404
      expr: sum(rate({kubernetes_namespace_name="test", kubernetes_pod_name=~"httpd.*"} |= "404" [1m])) > 0.1
      for: 10s
      labels:
        severity: critical
        cn: "{{ $labels.kubernetes_container_name }}"
      annotations:
        description: High 404 errors in httpd application
        summary: High 404 errors in httpd application

Version-Release number of selected component (if applicable):

Red Hat OpenShift Logging 5.8

Loki Operator 5.8

How reproducible:

100%

Steps to Reproduce:

1. Deploy clusterlogging and lokistack
2. Enable spec.rules so that ruler pods are deployed.
3. Create httpd application in a test project (oc new-app httpd) and expose the service to create the route. (oc expose svc/httpd)
4. Create an AlertingRule. Define a label in spec.groups.rules.label which uses parsed field. Say cn: "{{ $labels.kubernetes_container_name }}". Refer "Additional Info" section for the complete alert's YAML.
5. Let the alert to trigger.
6. Check if the label with parsed field appears in the alert.

Actual results:

The label defined with parsed field doesn't appear when the alert is triggered.

Expected results:

The label with parsed field should appear as per the requirement.

Additional info:

apiVersion: loki.grafana.com/v1
kind: AlertingRule
metadata:
  labels:
    app: httpd
  name: httpd-404-alerts
  namespace: test
spec:
  tenantID: application
  groups:
  - name: 404NotFound
    rules:
    - alert: TooMany404
      expr: sum(rate({kubernetes_namespace_name="test", kubernetes_pod_name=~"httpd.*"} |= "404" [1m])) > 0.1
      for: 10s
      labels:
        severity: critical
        cn: "{{ $labels.kubernetes_container_name }}"
      annotations:
        description: High 404 errors in httpd application
        summary: High 404 errors in httpd application

Run below command to met the above said condition.

$ while (true); do curl -s -o /dev/null -w "%{http_code}" http://httpd-test.apps.loki.example.com/abc/example; echo "\n";done

The httpd application doesn't have (/abc/example) path defined and above command streams 404 error in the application pod logs.