[Tech Debt] [Observability] Router doesn't verify the generated haproxy config on a per-route level

This is the continuation of https://bugzilla.redhat.com/show_bug.cgi?id=1861501.

Routes applied to a cluster with invalid annotations, etc can break reloads. When reloads are unsuccessful, newly admitted routes do not work. Additionally in this case, if a route's backends were to change (pod deletion, etc) that route may also break.

We need to discuss how we can better handle individual route configuration validation and or the failed state of router reloads resultant of a single route misconfiguration. Perhaps preliminary parsing of the HAProxy config file prior to a reload could help us detect when we would be performing a reload into a busted configuration (and somehow not apply the broken route?).

Further discussion needs to happen amonst the team in 4.9 planning time frame to pin down what we can do to alleviate this issue.