The OpenTelemetry collector now supports remote-write v2 receiver and it has been added to the RH build of OTEL. The use-case we would like to support is exporting the in-cluster/platform and UWM metrics to 3rd party vendors via OTLP (e.g. to Dynatrace).

To support this use-case customers should be able to configure in-cluster primetheus to remote-write to an OTEL collector and from the OTEL collector use OTLP exporter to 3rd party platform.

The OTEL remote-write receiver has a couple of requirements:

• only remote write V2
• enabled metadata-wal-records
• enabled native-histograms
• Enabled convertClassicHistogramsToNHCB

Here is the Prometheus CR with remote-write for sending data to the OTEL collector:

    convertClassicHistogramsToNHCB: true
    enableFeatures:
      - otlp-write-receiver
      - metadata-wal-records
      - native-histograms
    remoteWrite:
      - messageVersion: V2.0
        sendExemplars: true
        sendNativeHistograms: true
        url: http://otel-collector.observability.svc.cluster.local:9090/api/v1/write 

The objective of this ticket is to enable these settings for the in-cluster Prometheus.

POC with manifests: https://github.com/pavolloffay/openshift-observability-manifests/tree/main/export-metrics-from-ocp