Try finding optimizations on FLP metrics cache (TimedCache). This cache can be pretty huge and it basically duplicates what prom client stores internally (at least for keys).

Check if there are ways to re-use the internal registry.

In last resort, prom client can be forked to implement the desired expiration mechanism.

Check also with monitoring team if they're aware of anything that could help here.