Description of problem:

In production clusters with GPU functionality enabled by the IDH team, we regularly see the GPU entitlement become invalidated after ~2-3 weeks without warning. When this happens, GPUs cease to function whenever a cluster upgrade occurs or the RHODS CUDA notebook images need to be rebuilt.