This spike focuses on testing and refining the PromQL queries for the usage difference alerts. We will experiment with the provided expressions and explore variations to determine the most effective queries for identifying the intended scenarios without generating excessive false positives.

The scenarios to test are:

Metered usage exceeds tallied usage: swatch_metrics_ingested_usage_total / swatch_tally_tallied_usage_total > 1.0

Tally usage exceeds metered usage: swatch_tally_tallied_usage_total / swatch_metrics_ingested_usage_total > 1.0

Tallied usage exceeds billable/covered usage by greater than 1%: swatch_tally_tallied_usage_total / (swatch_contract_usage_total + swatch_billable_usage_total{status="succeeded"}) > 1.01

Billable and contract covered usage exceeds tallied usage by greater than 1%: (swatch_contract_usage_total + swatch_billable_usage_total{status="succeeded"}) / swatch_tally_tallied_usage_total > 1.01

Billable usage exceeds remitted usage: swatch_billable_usage_total{status="succeeded"} offset 1h / swatch_producer_metered_total > 1.0

Remitted usage exceeds billable usage: swatch_producer_metered_total / swatch_billable_usage_total{status="succeeded"} offset 1h > 1.0

Acceptance Criteria:

Create promql for each alert scenario

Test that the alert scenarios trigger accurately on real data.

Explore potential adjustments to the thresholds or query structure if needed.

Document the PromQL query that works best for each scenario.