Major Testing backend monitoring rules, I came accros some possible bug.
Deliverately I made backend listener to generate 500 errors, like the one in the log line:
10.130.1.181 - - [10/Jul/2020 16:55:35 UTC] "GET /transactions/authrep.xml?service_token=725f388b9a0c30d0f694628291b94721b7827daa75f318e5ae0bbe4fc4ea0b46&service_id=2&usage%5Bhits%5D=1&user_key=3cc1bd491350572e5aa7b4100c77d733 HTTP/1.1" 500 "Error connecting to Redis on backend-redis:6379 (Redis::TimeoutError)" 10.006187351 - "rejection_reason_header=1&limit_headers=1&no_body=1"
But, when requested metrics, there is no metrics for 5XX errors:
$ curl -v http://10.130.1.173:9394/metrics * Trying 10.130.1.173... * TCP_NODELAY set * Connected to 10.130.1.173 (10.130.1.173) port 9394 (#0) > GET /metrics HTTP/1.1 > Host: 10.130.1.173:9394 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Content-Type: text/plain; version=0.0.4 < Server: WEBrick/1.4.2 (Ruby/2.5.5/2019-03-15) < Date: Fri, 10 Jul 2020 16:55:13 GMT < Content-Length: 3393 < Connection: Keep-Alive < # TYPE apisonator_listener_response_codes counter # HELP apisonator_listener_response_codes Response codes apisonator_listener_response_codes{request_type="",resp_code="2xx"} 629.0 apisonator_listener_response_codes{request_type="authrep",resp_code="2xx"} 3211.0 apisonator_listener_response_codes{request_type="authrep",resp_code="403"} 24.0 # TYPE apisonator_listener_response_times_seconds histogram # HELP apisonator_listener_response_times_seconds Response times apisonator_listener_response_times_seconds_bucket{request_type="",le="0.01"} 619.0 apisonator_listener_response_times_seconds_bucket{request_type="",le="0.02"} 625.0 apisonator_listener_response_times_seconds_bucket{request_type="",le="0.03"} 628.0 apisonator_listener_response_times_seconds_bucket{request_type="",le="0.04"} 628.0 apisonator_listener_response_times_seconds_bucket{request_type="",le="0.05"} 628.0 apisonator_listener_response_times_seconds_bucket{request_type="",le="0.06"} 628.0 apisonator_listener_response_times_seconds_bucket{request_type="",le="0.07"} 629.0 apisonator_listener_response_times_seconds_bucket{request_type="",le="0.08"} 629.0 apisonator_listener_response_times_seconds_bucket{request_type="",le="0.09"} 629.0 apisonator_listener_response_times_seconds_bucket{request_type="",le="0.1"} 629.0 apisonator_listener_response_times_seconds_bucket{request_type="",le="0.25"} 629.0 apisonator_listener_response_times_seconds_bucket{request_type="",le="0.5"} 629.0 apisonator_listener_response_times_seconds_bucket{request_type="",le="0.75"} 629.0 apisonator_listener_response_times_seconds_bucket{request_type="",le="1"} 629.0 apisonator_listener_response_times_seconds_bucket{request_type="",le="+Inf"} 629.0 apisonator_listener_response_times_seconds_sum{request_type=""} 0.8301439940000002 apisonator_listener_response_times_seconds_count{request_type=""} 629.0 apisonator_listener_response_times_seconds_bucket{request_type="authrep",le="0.01"} 3195.0 apisonator_listener_response_times_seconds_bucket{request_type="authrep",le="0.02"} 3222.0 apisonator_listener_response_times_seconds_bucket{request_type="authrep",le="0.03"} 3227.0 apisonator_listener_response_times_seconds_bucket{request_type="authrep",le="0.04"} 3228.0 apisonator_listener_response_times_seconds_bucket{request_type="authrep",le="0.05"} 3229.0 apisonator_listener_response_times_seconds_bucket{request_type="authrep",le="0.06"} 3230.0 apisonator_listener_response_times_seconds_bucket{request_type="authrep",le="0.07"} 3232.0 apisonator_listener_response_times_seconds_bucket{request_type="authrep",le="0.08"} 3232.0 apisonator_listener_response_times_seconds_bucket{request_type="authrep",le="0.09"} 3232.0 apisonator_listener_response_times_seconds_bucket{request_type="authrep",le="0.1"} 3233.0 apisonator_listener_response_times_seconds_bucket{request_type="authrep",le="0.25"} 3234.0 apisonator_listener_response_times_seconds_bucket{request_type="authrep",le="0.5"} 3235.0 apisonator_listener_response_times_seconds_bucket{request_type="authrep",le="0.75"} 3235.0 apisonator_listener_response_times_seconds_bucket{request_type="authrep",le="1"} 3235.0 apisonator_listener_response_times_seconds_bucket{request_type="authrep",le="+Inf"} 3235.0 apisonator_listener_response_times_seconds_sum{request_type="authrep"} 10.123920129000002 apisonator_listener_response_times_seconds_count{request_type="authrep"} 3235.
Hence, alert is not raised.
