-
Story
-
Resolution: Done
-
Undefined
-
None
-
None
-
3
-
False
-
None
-
False
-
NetObserv - Sprint 222, NetObserv - Sprint 223, NetObserv - Sprint 224
FLP can expose metrics to prometheus without necessarily needing an aggregation stage. The prometheus client provides natively some aggregations:
- by defining dimensions (labels), data comes aggregated by these labels; so the metrics cardinality is unchanged, compared with a prior Aggregation stage
- data is "aggregated" in time between each scrape intervals. Counters add up, Histograms generate stats (percentiles/count/sum), only Gauges will loose in accuracy since only their value at scrape time are being recorded.
- note that, the "top-k" functionality of the Aggregation stage cannot be reproduced in the same way. Top-k can be performed at query-time with promQL, but an advantage of the pre-processed top-k offered by the aggregation stage is to reduce the amount of stored metrics, hence allowing a mitigation for high-cardinality labels.
- Some aggregations functions, like "count number of services", can be obtained via promql instead of being a metric by itself.
It seems like the metrics we're interested in for NetObserv operator don't need that aggregation stage most of the time, so we could remove it, thus optimizing FLP's pipeline
See also PoC https://github.com/netobserv/flowlogs-pipeline/pull/266
Note that, as said in the PoC, at the moment we must make a choice between having every metric going through aggregation stage, or having none. There's no in-between.
There is a follow-up task to make FLP able to mix metrics with and without pre-aggregation: NETOBSERV-498
- is related to
-
NETOBSERV-498 Make FLP able to mix metrics with or without pre-agg
- Closed
- links to
- mentioned on