Description of problem:

We currently have two customers reporting this issue where the thanos-store-shard PVCs are filling up, both are on ACM 2.5. Raising the storage to 50 GiB doesn't resolve the issue. Customer A has 17 managed clusters and Customer B has 216.

Customer A:

Currently at 15 GiB for thanos-store-shard
data-observability-thanos-store-shard-0-0 5.45 GiB
data-observability-thanos-store-shard-1-0 10.93 GiB
data-observability-thanos-store-shard-2-0 9.3.6 GiB
—
Customer B:

Attempted 50 GiB for thanos-store-shard but still ran out of space

/dev/sde 49G 49G 0 100% /var/thanos/store

We have currently 261 blocks in s3 bucket used by namacmgtd21p Thanos. 261 Blocks * 30MB = 7830 MB

Version-Release number of selected component (if applicable):

2.5.1, 2.5.2

How reproducible:

Consistent in customer environments

Steps to Reproduce:

1. Configure observability
2.  
3. ...

Actual results:

thanos-store-shard pvcs fill up

Expected results:

thanos-store-shard doesn't use significant amounts of local disk space as mentioned in the documentation.
"It acts primarily as an API gateway and therefore does not need significant amounts of local disk space. It joins a Thanos cluster on startup and advertises the data it can access. It keeps a small amount of information about all remote blocks on local disk and keeps it in sync with the bucket. "
"In general, an average of 6 MB of local disk space is required per TSDB block stored in the object storage bucket, but for high cardinality blocks with large label set it can even go up to 30MB and more. It is for the pre-computed index, which includes symbols and postings offsets as well as metadata JSON."

Additional info: