-
Bug
-
Resolution: Done
-
Blocker
-
None
-
None
-
None
-
Sprint 218
Due to https://issues.redhat.com/browse/MON-2467 we do not have applied rules from 25/04 14:00 until 28/04 11:15.
In order to backfill you will need to:
- Request temporary elevation of perms from app-sre (thread: https://coreos.slack.com/archives/CCRND57FW/p1651162201441069) to port-forward thanos-query in `observatorium-metrics-production` namespace in telemeter prod
- Download the telemeter rules from openshift/telemeter https://github.com/openshift/telemeter/blob/master/jsonnet/telemeter/rules.libsonnet
- Run promtool (https://github.com/prometheus/prometheus/tree/main/cmd/promtool) using the following command, pointing it at the port-forwarded production thanos-query
promtool tsdb create-blocks-from rules --start="2022-04-25T14:00:00.00Z" --end="2022-04-28T11:15:00.00Z" rules.yam
- You will then need to place the block in object storage where store-gateway can query them, this will need to be requested from app-sre as we keep our thanos-obj-store config in a secret in vault (path: app-interface/app-sre/telemeter-production/rules-objstore-objectstorage)
- Once placed, give a few minutes for store-gw to sync the block, and attempt to query the rules for the time range in info-gw
If everything works, the gap in this screenshot should disappear
_ _
- is related to
-
MON-2467 Telemeter rules are not being evaluated in production
-
- Closed
-