The following alerts is firing PrometheusDuplicateTimestamps

Turning on debug logs indicates that the noobaa-mgmt-service-monitor and s3-service-monitor are the issue

1 ts=2024-08-09T12:35:11.506Z caller=scrape.go:1777 level=debug component="scrape manager" scrape_pool=serviceMonitor/openshift-storage/noobaa-mgmt-service-monitor/0 target=http://10.129.2.15:8080/metrics/web_server msg="Duplicate sample for timestamp" series=NooBaa_health_status

This can be shown by manually curling the metrics

oc exec prometheus-k8s-1 – curl "http://10.129.2.15:8080/metrics/web_server" > metrics.txt

Which indeed is returning duplicate metrics by searching for 'NooBaa_health_status 0' as a example

Upon investigation this is because of this block
https://github.com/noobaa/noobaa-core/blob/ad73e9cb3bd483f6f34de9a28a9f4ba3ea060eb3/src/server/analytic_services/prometheus_reporting.js#L44

If I call /metrics/web_server/nodejs and /metrics/web_server/core seperatly they return the same results.

So the solution is to either alert the code above or change the service monitor items to append /nodejs onto the end.

Such as
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: noobaa-mgmt-service-monitor
labels:
app: noobaa
spec:
endpoints:

• port: mgmt
  path: /metrics/web_server/nodejs
• port: mgmt
  path: /metrics/bg_workers
• port: mgmt
  path: /metrics/hosted_agents
  namespaceSelector: {}
  selector:
  matchLabels:
  noobaa-mgmt-svc: "true"