We have an issue reported for the Prometheus pod (container: prometheus) consuming much memory.

Initially, the pod was in CrashLoop state getting OOMKilled with the default memory limit of 500Mi. We have increased the limit to 2Gi, still getting oom.

This has been observed in SMCP version 2.4. and the servicemeshoperator version is 2.4.5. There is only a single member namespace for this control plane with only 13 pods running. Consuming up to 2Gi mem doesn't seem normal given the size of the mesh.

The memory spike happens within 5 minutes of the prometheus pod restart and leads to oomkilled.

There is v2.3 SMCP as well in the same cluster without this issue.