Uploaded image for project: 'OpenShift Monitoring'
  1. OpenShift Monitoring
  2. MON-2473

Backfill Telemetry rules data due to stateless ruler outage

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Blocker Blocker
    • None
    • None
    • None
    • Sprint 218

      Due to https://issues.redhat.com/browse/MON-2467 we do not have applied rules from 25/04 14:00 until 28/04 11:15. 

      In order to backfill you will need to: 

      promtool tsdb create-blocks-from rules --start="2022-04-25T14:00:00.00Z" --end="2022-04-28T11:15:00.00Z" rules.yam
      • You will then need to place the block in object storage where store-gateway can query them, this will need to be requested from app-sre as we keep our thanos-obj-store config in a secret in vault (path: app-interface/app-sre/telemeter-production/rules-objstore-objectstorage) 
      • Once placed, give a few minutes for store-gw to sync the block, and attempt to query the rules for the time range in info-gw

       

      If everything works, the gap in this screenshot should disappear

      _

       

        1. image-2022-04-28-17-27-16-561.png
          image-2022-04-28-17-27-16-561.png
          269 kB
        2. image.png
          image.png
          269 kB
        3. screenshot-1.png
          screenshot-1.png
          111 kB
        4. screenshot-2.png
          screenshot-2.png
          110 kB

              ibillett@redhat.com Ian Billett (Inactive)
              mzardab@redhat.com Moad Zardab
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated:
                Resolved: