Description of problem:

• LabelKeys are missing at External Loki when using openshiftLabels filtering in CLF 

Version-Release number of selected component (if applicable)

• Logging 6.2, Logging 6.3, Logging 6.4

How reproducible:

Steps to Reproduce:

1. Install Red Hat OpenShift Logging operator v6.3.1
2. Create a ServiceAccount, Bind the Cluster Role, and Add additional roles to the collector service account:

$ oc -n openshift-logging create serviceaccount collector 
$ oc -n openshift-logging adm policy add-cluster-role-to-user logging-collector-logs-writer -z collector
$ oc -n openshift-logging adm policy add-cluster-role-to-user collect-application-logs -z collector   

1. Create a ClusterLogForwarder configuration.
2. Add a filter to insert openshift.labels.cluster_name in the log content
3. Make the added openshift label as a loki label key for query
4. ClusterLogForwarder configuration:

apiVersion: observability.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: instance
  namespace: openshift-logging
spec:
  filters:
  - name: my-labels
    openshiftLabels:
      cluster_name: cluster_4_20
    type: openshiftLabels
  managementState: Managed
outputs:
  - loki:
      labelKeys:
       - openshift.labels.cluster_name
       - log_type
       - kubernetes.container_name
       - kubernetes.namespace_name
       - kubernetes.pod_name
      tenantKey: $MY_LOKI_TENANT
      url: $MY_LOKI_URL
    name: my-loki
    type: loki
pipelines:
  - filterRefs:
    - my-labels
  inputRefs:
    - infrastructure
  name: infra-log-to-loki
  outputRefs:
    - my-loki
  - filterRefs:
    - my-labels
  inputRefs:
    - application
  name: app-log-to-loki
  outputRefs:
    - my-loki
serviceAccount:
  name: collector 

Actual results:

• At the Loki instance, only  "log_type" and "openshift.labels.cluster_name" are available as LabelKeys.
• But all kubernetes.* are not listed. And the fields are rewritten to be an empty string when we query from Grafana:
  "kubernetes": {         "container_name": "",         "namespace_name": "",         "pod_name": "" }

• The kubernetes.* fields in loki's label key are empty strings. Other kubernetes.* fields, like annotations, labels are not in the log result

Expected results:

• Below labels should be available in external Loki:

  k8s_container_name
  k8s_namespace_name
  k8s_node_name
  k8s_pod_name•
  kubernetes_container_name
  kubernetes_host
  kubernetes_namespace_name
  kubernetes_pod_name
  log_type
  openshift_log_type 

• The aim is to add a field "cluster_name" in the log content and extend it to the default loki label keys so that it can use the cluster_name to identify the log source in centralized loki.

Additional info: