What:
From RHOAM performance testing it seems like some connections fail whenever apicast pod is deleted. Most likely apicast is prematurely closing connections that are still in progress.
Additional investigation of the behavior is required. Perhaps one could create a slow connection to one of the apicast pods and observe if the connection is closed when pod is deleted, or whether apicast finishes the request before shutting down.

Why:
Connections fail with 5xx error when APICast pod is deleted.

How:
Perhaps these posts will be helpful:
https://blog.gruntwork.io/gracefully-shutting-down-pods-in-a-kubernetes-cluster-328aecec90d
https://blog.gruntwork.io/delaying-shutdown-to-wait-for-pod-deletion-propagation-445f779a8304

Expected behaviour:

When APIcast pod is terminated, it should gracefully shutdown

2024/06/28 03:55:08 [notice] 15#15: signal 3 (SIGQUIT) received from 46, shutting down
2024/06/28 03:55:08 [notice] 29#29: exiting
2024/06/28 03:55:08 [notice] 28#28: gracefully shutting down