Uploaded image for project: 'OpenShift Monitoring'
  1. OpenShift Monitoring
  2. MON-2095

Add validation on Thanos compactor of retention flags that does not make sense

XMLWordPrintable

    • Icon: Task Task
    • Resolution: Done
    • Icon: Undefined Undefined
    • None
    • None
    • Thanos
    • None
    • False
    • False
    • NEW
    • NEW
    • Monitoring - Sprint 213
    • 0

      This is a follow-up on a post mortem for:  https://docs.google.com/document/d/1XRjoL72S74_GrQ2WqkmvNIe4Tvy970MBc6dHzKr36CE/edit#heading=h.lzjvop3jozc5

      In this outage, we misconfigured our the `--retention.raw-resoluteion` was set to below the downsampled retention. This meant that thanos-compactor deleted the raw data before it could downsample it for the lower resolution blocks. 

      Goals: 

      • Investigate flag validation on thanos-compactor to stop this happening 
      • Investigate why the downsampling pass was not executed before the retention deletion

            mzardab@redhat.com Moad Zardab
            mzardab@redhat.com Moad Zardab
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: