-
Bug
-
Resolution: Unresolved
-
Normal
-
None
-
4.21.0
-
None
-
None
-
False
-
-
None
-
Moderate
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Description of problem:
configured prometheus externalLabels with testEnv=production, and remoteWrite writeRelabelConfigs labelmap action, the configuration renames the testEnv label to testingEnv for all metrics sent via remote_write, eliminating the original testEnv label and retaining only the new testingEnv label with its value.
see below
apiVersion: v1 data: config.yaml: | prometheusK8s: externalLabels: testEnv: production remoteWrite: - url: http://ip:8428/api/v1/write remoteTimeout: 30s writeRelabelConfigs: - action: labelmap regex: testEnv replacement: testingEnv kind: ConfigMap metadata: name: cluster-monitoring-config namespace: openshift-monitoring
the configuration loaded to prometheus
$ oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- cat /etc/prometheus/config_out/prometheus.env.yaml | grep -i remote_write -A20
remote_write:
- url: http://ip:8428/api/v1/write
remote_timeout: 30s
write_relabel_configs:
- target_label: __tmp_openshift_cluster_id__
replacement: e0b0422b-f625-40c5-a1d9-dc79b0510413
action: replace
- regex: testEnv
replacement: testingEnv
action: labelmap
- regex: __tmp_openshift_cluster_id__
action: labeldrop
should only see testingEnv=production for metrics in the remote write server, but see both testingEnv=production and testEnv=production
cluster_infrastructure_provider {container="kube-apiserver-operator",endpoint="https",instance="10.128.0.9:8443",job="kube-apiserver-operator",namespace="openshift-kube-apiserver-operator",pod="kube-apiserver-operator-65977984cf-m8tdv",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="metrics",testEnv="production",testingEnv="production",type="OpenStack"}
Version-Release number of selected component (if applicable):
4.21.0-0.nightly-2025-11-15-144034
How reproducible:
always
Steps to Reproduce:
1. see the description
Actual results:
see both testingEnv=production and testEnv=production
Expected results:
should only see testingEnv=production for metrics in the remote write server
Additional info:
