Custom notebooks - phase 2

Epic brief: https://docs.google.com/document/d/1YB6vlcwF0mG61CawEaAEsrR4a_18-2xFKwGwe1Anw0I/edit?usp=sharing

RHODS provides several out-of-the-box notebook images that automatically include packages to make it easy for users to get started with common components. However, in many/most instances, users need a specific set of packages/libraries with specific versions depending on the projects. There are so many possible variations of these packages/libraries/versions that it’s not practical to provide out-of-the-box images to anticipate the various needs. Users do not want to have to rely on pip installs for all the packages/libraries on a regular basis because it’s time consuming, and it can be difficult to keep track of all the needed components. They are looking for a quick & easy way to start a notebook session with the components they need for a project. Oftentimes, there can be repeatable patterns for the necessary components rather than starting from scratch for every project.

Many times users need specific package/library versions for a project, and they need to ensure that the combinations of packages & versions will work together. This could be driven by the need to leverage existing code that was created using a specific package/library version. They want to avoid scenarios where the notebook won’t run successfully because of an incompatibility across different packages & versions.

Requirements:

1. P0: Users must be able to create new notebook images from scratch (eg. starting with a Jupyter notebook rather than image) by defining the packages/libraries/components needed.

• Users would need specific permissions to make custom images available for all cluster users. This ensures some control over images that are available for selection for all cluster users. This can help ensure a good user experience.
• Starting from scratch means they could start by defining the specific Python version. 
• The UX should consider the preferred access point for users creating custom NB images - eg. from within JupyterLab.

2. P1: Users with specific permissions (other than admins) must be able to import custom images.

• Likely need to be separate group of users with explicit permissions to import custom NB images if they are available to all cluster users.

3. P0: Users must be able to start with an existing base notebook image (either provided by Red Hat or custom), make changes by adding/removing/updating components, and save the resulting image. The resulting image should be available for selection like other notebook images.

• Users would need specific permissions to make custom images available for all cluster users.

4. P1: Users must be able to make custom images private (only accessible by the user that created the custom image) or public (accessible to all cluster users).

• In order to make custom images public, users need explicit permissions - i.e. not all RHODS users should automatically be able to make custom NB images public.

5. P1: Users must be able to make custom images available only for one or more groups of users.

6. The system must provide users some capability to check dependencies and verify any dependency issues as they are defining the components in a notebook or a new custom image.

1.  P1: When users create custom NB images, the system should perform some automatic check for dependencies, security issues, and compatibility across components included in the image. The system should present a message with any issues found, but should not prevent users from proceeding. 
2. P2: The system should provide users an option to check for dependencies, security issues, and compatibility across components prior to attempting to create a custom NB image. This should be an optional tool accessible from the UI. The use case is enabling users to check for issues in a notebook/image prior to the process of trying to create a custom image.
3. P2: The system should provide optional guidance to identify dependencies (eg specific package versions needed) at the point of installing packages. For example, a user could add a TensorFlow version in a notebook and the user could access a tool to get recommendations on other package versions to install. Additionally, if a user installs an incompatible version of dependency (eg. the wrong version of scipy to work with the added version of TensorFlow), the user could access a UI option to identify issues and make corrections.  

Notes:

• Can Thoth service address this?
• This should be an optional tool that is available prior to creating a custom NB image. This could be at the point of installation or after I've installed several components & I want to check the versions & compatibility. 

7. P0: Admins must be able to export custom images & store on a local system.

Notes:

• This could be provided as part of standard OSD capabilities
• The idea is to give users an option to store custom images they create in a separate location for backup purposes.

8. P0: The system must provide some basic validation for uploaded and created custom images. The validation should ensure the custom image can be accessed & run in the RHODS JupyterLab environment.

• Note: this requirement is not about checking dependencies across components within a package. That is covered in req 6(1) above.