Uploaded image for project: 'Red Hat OpenShift Data Science'
  1. Red Hat OpenShift Data Science
  2. RHODS-4934

Unable to spawn notebook server with Tensorflow, PyTorch and CUDA images

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Blocker Blocker
    • RHODS_1.15.0_GA
    • None
    • Workbenches
    • None
    • False
    • None
    • False
    • Release Notes
    • Yes
    • Yes
    • Hide
      == Jupyter failed to start TensorFlow, PyTorch, or CUDA notebook servers intermittently
      Jupyter's *Start a notebook server* page intermittently failed to start a notebook server using the TensorFlow, PyTorch and CUDA notebook images. These notebook images now successfully start without intermittent failures.
      Show
      == Jupyter failed to start TensorFlow, PyTorch, or CUDA notebook servers intermittently Jupyter's *Start a notebook server* page intermittently failed to start a notebook server using the TensorFlow, PyTorch and CUDA notebook images. These notebook images now successfully start without intermittent failures.
    • Bug Fix
    • Done
    • Yes
    • Yes
    • None
    • Critical

      Description of problem:

      Unable to spawn jupyterhub notebook server with Tensorflow, Pytorch and CUDA image.

      Getting "Back-off restarting failed container" error.

      Prerequisites (if any, like setup, operators/versions):

      Steps to Reproduce

      1. Install RHODS 1.15.0-10
      2. Spawn a notebook server with Tensorflow image

      Actual results:

      Getting "Back-off restarting failed container" error. Notebook container pod goes to CrashLoopBackOff state.

      The pod logs shows this message that seems to be the source of the problem:

      /opt/app-root/bin/oc: line 12: /opt/app-root/bin/oc-3.11: No such file or directory

      Expected results:

      Notebook server with Tensorflow image should spawn successfully.

      Reproducibility (Always/Intermittent/Only Once):

      Always

      Build Details:

      RHODS v1.15.0-10

      Workaround:

      Additional info:

        1. jupyterhub-nb-ldap-2dadmin1-notebook.log
          2 kB
        2. Screenshot from 2022-08-18 14-11-01.png
          Screenshot from 2022-08-18 14-11-01.png
          42 kB
        3. Screenshot from 2022-08-18 14-12-39.png
          Screenshot from 2022-08-18 14-12-39.png
          102 kB

              svelosol@redhat.com Samuel Veloso (Inactive)
              rhn-aloganat Arthy Loganathan
              Arthy Loganathan Arthy Loganathan
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

                Created:
                Updated:
                Resolved: