Description of problem:

In the Internal Data Hub instance of RHODS we have recently been seeing JupyterHub get into a broken state where any user sees an "unable to load notebook server configuration options" error message on the spawner page. (See attached screenshot)

I suspect that there may be an issue with our OpenShift cluster that is causing this, so I don't think it's necessarily a critical RHODS bug. Currently, the only way we've found to recover from this is to manually kill the JupyterHub leader pod (i.e. restart the service). I think this is a failure scenario that our liveness probes need to detect and automatically recover from.

Prerequisites (if any, like setup, operators/versions): 

Not known.

Steps to Reproduce

Not reproducible, but it is consistently occurring on the internal data hub cluster

Actual results:

Jupyterhub unexpectedly enters a failed state.

Expected results:

Jupyterhub always works.

Reproducibility (Always/Intermittent/Only Once):

Not reproducible, but it is consistently occurring on the internal data hub cluster

Build Details:

RHODS 1.9.0-7 installed via the OLMInstall repo

Workaround:

Restart jupyterhub (i.e. kill the leader pod)

Additional info:

None