Uploaded image for project: 'Hawkular Metrics'
  1. Hawkular Metrics
  2. HWKMETRICS-606

Large (> 100 MB) partitions in metrics_idx table make Cassandra unstable

This issue belongs to an archived project. You can view it, but you can't modify it. Learn more

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: 0.18.0, 0.19.0, 0.20.0, 0.21.0, 0.22.0, 0.23.0, 0.24.0
    • Fix Version/s: 0.21.8, 0.23.8, 0.26.1
    • Component/s: Core
    • Labels:
      None

      Description

      I have looked at some logs from a couple different environments that have warnings like this:

      WARN  18:29:53 Writing large partition hawkular_metrics/metrics_idx:ops-health-monitoring:2 (485941805 bytes)
      

      That warning is telling us we are writing a 496 MB partition to disk. The warning is configurable via the compaction_large_partition_warning_threshold_mb property in cassandra.yaml. It defaults to 100 MB. Overly large partitions can result in a lot of heap pressure during compaction as that partition is loaded into to memory from each of the SSTables in which it is found.

      The net effect is that client requests to Hawkular Metrics will start to fail when large partitions like this are being compacted. We are going to have to replace metrics_idx with a table that does some sharding to help prevent overly large partitions.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              john.sanda John Sanda (Inactive)
              Reporter:
              john.sanda John Sanda (Inactive)
              Archiver:
              ranumula Raju Anumula

                Dates

                Created:
                Updated:
                Resolved:
                Archived: