-
Bug
-
Resolution: Won't Do
-
Critical
-
ACM 2.5
-
False
-
None
-
False
-
-
-
-
-
No
Description of problem:
We currently have two customers reporting this issue where the thanos-store-shard PVCs are filling up, both are on ACM 2.5. Raising the storage to 50 GiB doesn't resolve the issue. Customer A has 17 managed clusters and Customer B has 216.
Customer A:
Currently at 15 GiB for thanos-store-shard
data-observability-thanos-store-shard-0-0 5.45 GiB
data-observability-thanos-store-shard-1-0 10.93 GiB
data-observability-thanos-store-shard-2-0 9.3.6 GiB
—
Customer B:
Attempted 50 GiB for thanos-store-shard but still ran out of space
/dev/sde 49G 49G 0 100% /var/thanos/store
We have currently 261 blocks in s3 bucket used by namacmgtd21p Thanos. 261 Blocks * 30MB = 7830 MB
Version-Release number of selected component (if applicable):
2.5.1, 2.5.2
How reproducible:
Consistent in customer environments
Steps to Reproduce:
- Configure observability
- ...
Actual results:
thanos-store-shard pvcs fill up
Expected results:
thanos-store-shard doesn't use significant amounts of local disk space as mentioned in the documentation.
"It acts primarily as an API gateway and therefore does not need significant amounts of local disk space. It joins a Thanos cluster on startup and advertises the data it can access. It keeps a small amount of information about all remote blocks on local disk and keeps it in sync with the bucket. "
"In general, an average of 6 MB of local disk space is required per TSDB block stored in the object storage bucket, but for high cardinality blocks with large label set it can even go up to 30MB and more. It is for the pre-computed index, which includes symbols and postings offsets as well as metadata JSON."
Additional info:
- clones
-
ACM-2674 Thanos Store persistent volumes filling up
- Closed