The scale & perf team did some experiments a while ago but nothing beats live data from customers so it would be interesting to examine what we have in telemetry and try to come up with rough sizing guidelines

e.g. with x nodes and y pods, Prometheus can scrape x samples/s and store y "live" time series which translates to z amount of RAM

Slack conversation: https://coreos.slack.com/archives/C0VMT03S5/p1637763209393400

DoD:

• Evaluate whether we have enough data points in telemetry to figure out the answer for above question
• Evaluate the possibilities to come up with a dashboard for monitoring team with the above data