Improve Kiali on Prometheus Production scenarios

This EPIC is related to OSSM-566.  Progress on OSSM-566 would impact this Epic, but parts of this Epic do not require OSSM-566.

Today OpenShift ServiceMesh defines a deployment model where Prometheus, Grafana, Kiali instances are co-located with the mesh control plane instance.

If as result of OSSM-566, the previous model may change, that may impact Kiali model as well.

It's expected that at some point OSSM will delegate the integration of Prometheus with the instance deployed into the cluster.

That scenario may be an opportunity for Kiali in the following areas:

• A shared instance of Prometheus between multiple (federated) meshes may improve the federation telemetry, then Kiali would improve the visibility of these scenarios within a single cluster.
• If OSSM integrates with addons that are deployed at cluster instance level (Prometheus, Grafana, Jaeger), that may trigger a discussion if Kiali could also benefit to be located at instance level .

Depending of the progress of OSSM-566, some questions should be answered from Kiali perspective:

• Assuming that scope of this task is single cluster (multi-cluster would have different effort).
• If Prometheus instance is located per instance cluster instead control plane, Kiali could be deployed per instance cluster as well.
• A Kiali instance would be able to connect with multiple control planes deployed within same cluster.
• A single Kiali instance may show end-to-end federated scenarios within same cluster.

Independent of OSSM-566, this Epic involves the following:

• Establishing a performance test environment that allows studying Kiali-Prometheus interactions at scale.
• Providing documentation, and possibly configurations, to improve performance via:
  • metric thinning (wthout degrading kiali)
    • complete removal of certain metrics
  • attribute thinning (without degrading kiali)
    • removal of certain attributes on certian metrics
  • metric crippling (with understood degradation of kiali)
    • removal of metrics and/or attributes
  • prometheus tuning (with understood impacts on kiali)

 Github Issue
https://github.com/kiali/kiali/issues/5151 Thinning Done