Description of problem:
It is not possible to scale down GPUs in a notebook if all the GPUs instance are occupied.
This happens regardless if the GPUs are occupied by the same workbench or others.
Prerequisites (if any, like setup, operators/versions):
Create a DS Project
Deploy a GPU node (1 GPU used to reproduce this bug)
Steps to Reproduce
- create and start a WB with 1 GPU
- stop it
- create a new WB with 1 GPU and launch it (you may need to wait a bit until the GPUs becomes available again after step 2)
- edit the first WB to scale the GPU from 1 to 0 and launch it
or
- create and start a WB with 1 GPU
- edit the WB to scale down the GPUs
*the issue should have a bigger impact in presence of multiple workbenches with multiple GPUs
Actual results:
it's not allowed to scale to 0 the GPU in order to launch the workbench
Expected results:
possible to scale down to 0 GPUs of a workbench
Reproducibility (Always/Intermittent/Only Once):
Always
Build Details:
RHODS v1.22.0-2
Workaround:
- Stop all the running workbenches which are using the GPUs
- wait until the GPU availability is captured
- edit the workbench and scale down the GPU instances