Uploaded image for project: 'Red Hat OpenShift Data Science'
  1. Red Hat OpenShift Data Science
  2. RHODS-4769

GPU drop down not showing up when GPU nodes are tainted.

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Normal
    • None
    • None
    • Integrations, UI
    • False
    • None
    • False
    • Release Notes
    • No
    • No
    • GPUs on nodes any taint other than the supported 'nvidia.com/gpu' taint cannot be selected when creating a notebook server. To avoid this issue, use only the nvidia.com/gpu taint on GPU nodes used with OpenShift Data Science.
    • Known Issue
    • Done
    • No
    • Pending
    • None

    Description

      Description of problem:

      GPU drop-down is not showing up. I thought maybe someone was already using the GPU. that was not the case. 

      I have one GPU node, and there was no Notebook pods on it. 

      I have a feeling I might know why. Hold on. I need to test my theory. 

      So, yeah. The difference is that I usually add a Taint to my nodes called "notebooksonly". I did add it to the GPU node, because:

      A) that node has no taint by default

      B) I had noticed random pods landing on it and hogging it for nothing

      However, as soon as I add the taint, a few minutes later, the drop down disappears. 

      When I remove the taint, the drop-down re-appears. 

       

      Prerequisites (if any, like setup, operators/versions):

      RHODS + GPU

      Steps to Reproduce

      1. add a taint called "notebooksonly" to the GPU machine pool. 
      2. wait 5 minutes
      3.  
      4.  

      Actual results:

      drop down no longer shows up in JupyterHub. 

      Expected results:

      I'd expect it to still show. 

      Reproducibility (Always/Intermittent/Only Once):

      Always

      Build Details:

      Workaround:

      None

      Additional info:

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              egranger@redhat.com Erwan Granger
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: