Investigate the issue where pod health checks do not appear to be working. During the recent 502 incident, a single pod was generating all of the 502 errors and was still receiving traffic but not being killed by the health check. If we manually killed the pod, 502s would move to a different pod a few hours later. Perform a thorough investigation to identify the root cause and propose solutions.