-
Bug
-
Resolution: Done
-
Major
-
OSSM 2.6.2
-
False
-
None
-
False
-
-
When the `UntilSuccess` method fails after the timeout/number of attempts is reached, the test continues to be executed.
Imagine a test case that tests 5 similar values in a row and a generalized submethod uses `UntilSuccess`.
func testBookinfoAppReportIstioRequestsTotal(t test.TestHelper) {
t.LogStep("Test that the `istio_requests_total` metric exist for bookinfo")
waitUntilPrometheusTargetReady(t, "productpage")
// so the test should fail right now and not execute the next checking
generateBookinfoTraffic(t)
checkIstioRequestsTotalInPrometheus(t, "productpage")
checkIstioRequestsTotalInPrometheus(t, "details")
checkIstioRequestsTotalInPrometheus(t, "reviews")
checkIstioRequestsTotalInPrometheus(t, "ratings")
}
func checkIstioRequestsTotalInPrometheus(t test.TestHelper, app string) {
retry.UntilSuccess(t, func(t test.TestHelper) {
oc.Exec(...)
})
}
func waitUntilPrometheusTargetReady(t test.TestHelper, app string) {
t.LogStep(`Wait till targets are available in Prometheus"`)
retry.UntilSuccess(t, func(t test.TestHelper) {
oc.Exec(...)
})
}
It is unnecessary to run all of them when the first fails ( since for each of them, the test executions spend some time before failing) (by default, UntilSuccess takes 1 minute to fail, so unnecessary 5 minutes) which increases 3 times because we rerun unsuccessful test 3 times. ( so after all, unnecessary 15 minutes )
e.g. example of execution the test above (with failure) with `FailNow` after the first failure and without ( the test execution was* 3x faster* )
====== Test summary v2.6: DONE 3 runs, 3 tests, 3 failures in 521.532s real 8m48.437s user 2m24.685s sys 0m57.969s ====== Test summary v2.6: DONE 3 runs, 3 tests, 3 failures in 1714.594s real 28m41.628s user 7m48.229s sys 3m2.420s