Uploaded image for project: 'Red Hat Advanced Cluster Management'
  1. Red Hat Advanced Cluster Management
  2. ACM-27042

Remove the compaction level limitation on the compactor and replace it with max-index-size

XMLWordPrintable

    • False
    • Hide

      None

      Show
      None
    • False
    • Not Selected
    • None

      Context:

      In the following PR, the compaction max level was reduced to 3. The intent was probably to reduce the index sizes in deployments with high cardinality.
      However, the downside of this setting is that it prevents the compactor from building 1h downsampled blocks in lower cardinality setups. The compactor indeed checks the blocks time span before compacting it. The current implementation checks for a block length higher than 10 days. In compaction levels terms, 3 generates 2 days blocks. And level 4 generate 14 days blocks (the default).
      The solution would be to remove this level limitation flag and replace it with the 'compact.block-max-index-size' one. It would not limit downsampling on most setups while keeping manageable index sizes on bigger setups.

      We end up in a situation where the raw resolution is 5m if the standard downsampling value is used, the 5m downsampling is useless and bigger than the raw one as it adds some statistics data, and the 1h downsampled block can't be built...

      Acceptance:

      The level limitation flag is removed and replaced with the index size limitation one. The correct value must be defined with the team.

              rh-ee-dbuchana Daniel Buchanan
              rh-ee-tmange Thibault Mange
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: