-
Bug
-
Resolution: Won't Do
-
Undefined
-
None
-
4.11.z
-
Moderate
-
No
-
False
-
Description of problem:
[bz-monitoring][invariant] alert/Watchdog must have no gaps or changes fails in CI. The error is:
[bz-monitoring][invariant] alert/Watchdog must have no gaps or changes expand_less 0s
{ Watchdog alert had 51187 changes during the run, which may be a sign of a Prometheus outage in violation of the prometheus query SLO of 100% uptime
Oct 12 03:38:35.795 - 2762s E alert/Watchdog ns/openshift-monitoring ALERTS{alertname="Watchdog", alertstate="firing", namespace="openshift-monitoring", prometheus="openshift-monitoring/k8s", severity="none"}
Oct 12 04:25:08.795 - 4111s E alert/Watchdog ns/openshift-monitoring ALERTS{alertname="Watchdog", alertstate="firing", namespace="openshift-monitoring", prometheus="openshift-monitoring/k8s", severity="none"}}
The job link is https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-nightly-4.11-e2e-aws-upgrade-rollback-oldest-supported/1712303574911815680
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
