(Slack thread / redhat internal: https://redhat-internal.slack.com/archives/C0VMT03S5/p1715862728898369)

It requires upstream changes to https://github.com/prometheus-community/prom-label-proxy  , which currently is used in the monitoring stack and configured to map the "namespace" query param to label enforcement. Our metrics don't have the "namespace" label but they have src/dest namespaces.

So we need:

• an upstream change to make prom-label-proxy support multiple labels config (it must allow mapping "src_namespace", while still continue to map "namespace" for the monitoring' nominal use case)
• update the cluster-monitoring-operator config to allow using this config

This is a bit tricky as it makes CMO having specific configs for Netobserv, regardless if netobserv is actually installed or not ( ? )

Also if this is tied to the CMO it means the feature will be only available in newest OCP.

This should be coordinated with the monitoring team.

There is actually an upstream issue to track this: https://github.com/prometheus-community/prom-label-proxy/issues/190

—

For now we implemented a workaround using metric relabel:

                    MetricRelabelConfigs: []*monitoringv1.RelabelConfig{
                        {
                            SourceLabels: []monitoringv1.LabelName{"__name__", "DstK8S_Namespace"},
                            Separator:    ptr.To("@"),
                            Regex:        "netobserv_(workload|namespace)_ingress_.*@(.*)",
                            Replacement:  "${2}",
                            TargetLabel:  "namespace",
                            Action:       "replace",
                        },
                        {
                            SourceLabels: []monitoringv1.LabelName{"__name__", "SrcK8S_Namespace"},
                            Separator:    ptr.To("@"),
                            Regex:        "netobserv_(workload|namespace)_(egress|flows|drop|rtt|dns)_.*@(.*)",
                            Replacement:  "${3}",
                            TargetLabel:  "namespace",
                            Action:       "replace",
                        },
                    },

This seems to be good enough as we pick corresponding metric either from Source or Destination according to regexes; however, that could be improved exposing both as admin view implementing the solution above