Proposed title of this feature request

Reduce scraping impact on performance

What is the nature and description of the request?

Currently, IBM Z customers are requesting to reduce IFL(Integrated Facility for Linux) consumption in clusters due to scraping rates. However, it most likely makes sense to not scope this to a single architecture, if there is no reason for it.

Prometheus scraping rate is the interval node-exporter is gathering data from nodes and send it over to Prometheus. Data scraping is done on default every 30seconds. However, scraping of data is an IFL and memory intensive task which scales with the number of compute nodes, pods, and other resources etc.

The proposal is to make the scraping rate configurable, so that customers can control the impact of scraping on the performance.

The default scraping configuration and behavior is not supposed to change by this, rather, this should introduce an additional optional config flag to set this to levels, potentially only to a fixed set of values.

According to a RH blog article changing the scraping rate has potential of 30% in relaxed clusters and up to factor 2-4 (depending on scraping rate) in heavy used clusters by reducing Prometheus and kubelet CPU requests.

https://www.redhat.com/en/blog/openshift-monitoring-stack-playing-with-prometheus-performance-and-scraping-intervals

Why does the customer need this? (List the business requirements)

On IBM Z, the use case is to reduce IFL consumption on Cluster Level to lower Steady State.

List any affected packages or components.

[Please describe]