Uploaded image for project: 'Red Hat Advanced Cluster Management'
  1. Red Hat Advanced Cluster Management
  2. ACM-11577

Thanos Compactor stuck causing Store Gateways to run out of disk

XMLWordPrintable

    • False
    • None
    • False
    • Customer Facing, Customer Reported
    • No

      Slack thread: https://redhat.enterprise.slack.com/archives/CUU609ZQC/p1714773066996089

      Description of problem:

      The customer's Compactor got stuck with errors like:

      ts=2024-05-06T12:33:44.645959333Z caller=compact.go:491 level=error msg="critical error detected; halting" err="compaction: group 0@14678226527745613666: compact blocks [/var/thanos/compact/compact/0@14678226527745613666/01HMA15WHE9N1SKAENZRXTHD1X /var/thanos/compact/compact/0@14678226527745613666/01HMA15WHDX0BRNJ5JR17JWSD0]: populate block: chunk iter: cannot populate chunk 8 from block 01HMA15WHDX0BRNJ5JR17JWSD0: segment index 0 out of range"

      Which left it stuck for a long time, causing the Store GWs to run out of disk storage.  

      Additional info: https://access.redhat.com/support/cases/03807207

              rh-ee-doolivei Douglas Camata (Inactive)
              rh-ee-doolivei Douglas Camata (Inactive)
              Xiang Yin Xiang Yin
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: