Uploaded image for project: 'Red Hat OpenShift Data Science'
  1. Red Hat OpenShift Data Science
  2. RHODS-1897

Some older versions of libraries cannot be installed in JupyterLab (multiple images)

XMLWordPrintable

    • MODH Sprint 31

      Description of problem:

      When trying to install (some) older versions of (some) libraries in any image provided in the JupyterHub spawner, the installation will fail.
      The issue only seems to arise when pip cannot fetch a .wheel directly but has to build one through a setup.py file, and the error message seems to always involve gcc (i.e. only libraries that include dependencies that need to be compiled). However, the problem does not happen every time pip cannot fetch a .wheel directly.

      Scenarios which could lead to this bug:

      1) Data Analyst(s) working in self-contained notebooks that pull specific library versions or do blind installs

      • Data Analyst maintains an ETL or other pipeline locally to analyze some data in jupyter notebooks
      • S/he does not maintain a requirements file but simply install libraries system-wide/in a virtual env
      • Uploads the notebook to RHODS expecting to be able to run their code.
        • Might install an older version of a library they use because it's a known good version, or do a blind install of latest, which might result in breakage or weird behaviour leading to the DA trying to install an older version of said library
      • DA encounters the bug

      2) Data Science team in customer org maintaing a reasonably big and complex ML/AI project, using `requirements.txt` (or equivalent) files to have a consistent environment with known good versions of libraries

      • DS team decides to move the project to the cloud (RHODS)
      • They set everything up in one of the images, and start installing their dependencies
        • `pip install -r requirements.txt` or similar
      • One of the libraries at the pinned version in the reqs file fails to install
      • DS Team encounters the bug

      Prerequisites (if any, like setup, operators/versions):

      RHODS 1.1.1-29 on OSD

      Steps to Reproduce

      1. Spawn a JH image
      2. Try installing an older version of a library via pip (right now confirmed scikit-learn==0.15, pandas==0.25 and numpy==1.9 on minimal python and tensorflow images)

      Actual results:

      Installation fails when building the wheel

      Expected results:

      Library is installed at the specified version

      Reproducibility (Always/Intermittent/Only Once):

      Always

      Build Details:

      RHODS 1.1.1-29 on OSD

      Additional info:

      Logs are too long to paste here, attaching an .ipynb file with the error messages.

            jkoehler@redhat.com Jacqueline Koehler
            rhn-support-lgiorgi Luca Giorgi
            Jeff DeMoss
            Luca Giorgi Luca Giorgi
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: