Uploaded image for project: 'Red Hat Advanced Cluster Management'
  1. Red Hat Advanced Cluster Management
  2. ACM-3745

Thanos Store persistent volumes filling up(2.6)

XMLWordPrintable

    • False
    • None
    • False
    • -
    • No

      Description of problem:

      We currently have two customers reporting this issue where the thanos-store-shard PVCs are filling up, both are on ACM 2.5. Raising the storage to 50 GiB doesn't resolve the issue. Customer A has 17 managed clusters and Customer B has 216.

      Customer A:

      Currently at 15 GiB for thanos-store-shard
      data-observability-thanos-store-shard-0-0 5.45 GiB
      data-observability-thanos-store-shard-1-0 10.93 GiB
      data-observability-thanos-store-shard-2-0 9.3.6 GiB

      Customer B:

      Attempted 50 GiB for thanos-store-shard but still ran out of space

      /dev/sde 49G 49G 0 100% /var/thanos/store

      We have currently 261 blocks in s3 bucket used by namacmgtd21p Thanos. 261 Blocks * 30MB = 7830 MB

      Version-Release number of selected component (if applicable):

      2.5.1, 2.5.2

      How reproducible:

      Consistent in customer environments

      Steps to Reproduce:

      1. Configure observability
      2.  
      3. ...

      Actual results:

      thanos-store-shard pvcs fill up

      Expected results:

      thanos-store-shard doesn't use significant amounts of local disk space as mentioned in the documentation.
      "It acts primarily as an API gateway and therefore does not need significant amounts of local disk space. It joins a Thanos cluster on startup and advertises the data it can access. It keeps a small amount of information about all remote blocks on local disk and keeps it in sync with the bucket. "
      "In general, an average of 6 MB of local disk space is required per TSDB block stored in the object storage bucket, but for high cardinality blocks with large label set it can even go up to 30MB and more. It is for the pre-computed index, which includes symbols and postings offsets as well as metadata JSON."

      Additional info:

              Unassigned Unassigned
              rhn-support-rspagnol Ryan Spagnola
              Xiang Yin Xiang Yin
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: