Details
-
Task
-
Resolution: Duplicate
-
Undefined
-
None
-
None
-
False
-
-
False
Description
imported from original jira issue https://issues.redhat.com/browse/RHODS-7570
Description of problem:
After installing ods operator, we can see this warnings in prometheus pod logs in openshift-monitoring namespace:
ts=2023-03-13T17:27:44.783Z caller=scrape.go:1649 level=warn component="scrape manager" scrape_pool=serviceMonitor/redhat-ods-monitoring/modelmesh-federated-metrics/0 target="https://10.128.2.14:9091/federate?match%5B%5D=%7B_name%3D+%22haproxy_backend_http_average_response_latency_milliseconds%22%7D&match%5B%5D=%7Bname_%3D+%22haproxy_backend_http_responses_total%22%7D" msg="Error on ingesting out-of-order samples" num_dropped=146
ts=2023-03-13T17:28:14.784Z caller=scrape.go:1649 level=warn component="scrape manager" scrape_pool=serviceMonitor/redhat-ods-monitoring/modelmesh-federated-metrics/0 target="https://10.128.2.14:9091/federate?match%5B%5D=%7B_name%3D+%22haproxy_backend_http_average_response_latency_milliseconds%22%7D&match%5B%5D=%7Bname_%3D+%22haproxy_backend_http_responses_total%22%7D" msg="Error on ingesting out-of-order samples" num_dropped=146
Prerequisites (if any, like setup, operators/versions):
ods operator version: 1.22.1-4
openshift version 4.12
it's also reproducible in 4.10
Workaround:
remove redhat-ods-monitoring from being scrapped by prometheus:
oc label namespace redhat-ods-monitoring openshift.io/cluster-monitoring-