We are seeing a sporadic prometheus runtime recording rule evaluation failure.

ts=2023-03-02T19:27:58.773Z caller=manager.go:634 level=warn component="rule manager" file=/etc/prometheus/rules/prometheus-obs-prometheus-rulefiles-0/managed-application-services-observability-kafka-recording-rules.yaml group=kafka-recording-rules name=kube_node_labels_condition index=1 msg="Evaluating rule failed" rule="record: kube_node_labels_condition\nexpr: kube_node_labels * on(node) group_left(condition) kube_node_status_condition\nlabels:\n observability: managed-kafka-staging\n" err="found duplicate series for the match group

on the right hand-side of the operation: [{__name__=\"kube_node_status_condition\", condition=\"Ready\", container=\"kube-rbac-proxy-main\", endpoint=\"https-main\", job=\"kube-state-metrics\", namespace=\"openshift-monitoring\", node=\"ip-10-0-131-5.ec2.internal\", prometheus=\"openshift-monitoring/k8s\", prometheus_replica=\"prometheus-k8s-1\", service=\"kube-state-metrics\", status=\"true\"}, {__name__=\"kube_node_status_condition\", condition=\"Ready\", container=\"kube-rbac-proxy-main\", endpoint=\"https-main\", job=\"kube-state-metrics\", namespace=\"openshift-monitoring\", node=\"ip-10-0-131-5.ec2.internal\", prometheus=\"openshift-monitoring/k8s\", prometheus_replica=\"prometheus-k8s-0\", service=\"kube-state-metrics\", status=\"true\"}];many-to-many matching not allowed: matching labels must be unique on one side"