Whenever one of the mecached pods disappears (because of a rolling restart during a minor update or as result of a failure) APIs take a long time to detect that the pod went away and keep trying to reconnect.

From a quick round of tests we saw that the API downtime was ~150s or so.

Looking at the services (mainly keystone and nova-api) config files we saw that there are a couple of parameters that could be useful:

enable_retry_client=true
retry_attempts=X
retry_delay=Y

We tested with retry_attempts=2 and retry_delay=0 and the APIs recovered much faster.