Volume unmount repeats after successful unmount, preventing pod delete

We have observed a situation where:
- A workload mounting multiple EBS volumes gets stuck in a Terminating state when it finishes.
- The node that the workload ran on eventually gets stuck draining, because it gets stuck on unmounting one of the volumes from that workload, despite no containers from the workload now running on the node.

What we observe via the node logs is that the volume seems to unmount successfully. Then it attempts to unmount a second time, unsuccessfully. This unmount attempt then repeats and holds up the node.

Specific examples from the node's logs to illustrate this will be included in a private comment. 

4.11.5

Has occurred on four separate nodes on one specific cluster, but the mechanism to reproduce it is not known.

1.
2.
3.

A volume gets stuck unmounting, holding up removal of the node and completed deletion of the pod.

The volume should not get stuck unmounting.